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Abstract 

Concept  selection  is  an  important  investment  decision  point  in  product  develop¬ 
ment.  The  Department  of  Defense  too  often  selects  concepts  based  on  insufficient  data, 
resulting  in  projects  that  are  over-budget,  over-schedule,  and  not  what  the  customer 
wants.  Decision  makers  must  select  for  further  development  only  the  concepts  that  are 
effective  and  suitable  to  meet  the  needs  of  the  users  and  require  mature  concepts  to  make 
an  infonned  decision.  This  research  proposes  a  stage-gated  framework  as  a  tool  to  assess 
and  increase  the  maturity  of  concepts  by  creating  an  information  criteria  baseline  at  each 
decision  gate.  The  framework  represents  a  developmental  scale  that  allows  a  concept  to 
be  evaluated  relative  to  its  phase  of  development  rather  than  to  a  complete  material  con¬ 
cept.  The  infonnation  criteria  for  each  gate  are  derived  from  a  four-step  process  using 
well-understood  systems  engineering  and  architecture  principles  that,  when  combined  at 
early  decision  points,  provide  the  right  level  of  information  at  the  right  time.  It  is  antic¬ 
ipated  that  the  proposed  stage-gated  maturity  framework  will  provide  a  useful  tool  to 
practitioners  and  decision  makers  involved  in  the  development  of  concepts. 


AFIT/GRD/ENV/1 0-M05 


Dedication 


To  my  amazing  wife. 


-  v  - 


Acknowledgements 


I  would  like  to  thank  my  research  partner,  Capt  Abe  Barker,  for  undertaking  this 
effort  with  me.  His  hard  work,  dedication  and  good  humor  made  this  project  much  more 
enjoyable.  I  would,  also,  like  to  thank  my  research  advisors,  Lt  Col  Robb  Wirthlin  and 
Dr.  David  Jacques  for  their  instruction  and  guidance  regarding  the  research  topic  and  the 
academic  research  process,  they  were  always  there  to  remind  us  that  humor  has  no  place 
in  academic  writing.  Like  good  prison  guards,  they  kept  Abe  and  I  in-line  and  helped  us 
get  out  alive.  For  that,  I  am  eternally  grateful. 

Additionally,  I  would  like  to  thank  my  bosses  and  colleagues  at  Los  Angeles  Air 
Force  Base.  Specifically,  I  thank  Col  Bill  Harding  and  Mr.  Luke  Schaub  for  giving  me 
the  opportunity  to  work  in  challenging  jobs.  I  thank  Col  Sara  Beth  Cliatt,  Lt  Col  Chris 
Beverly,  and  Maj  John  Mizell  for  their  guidance  and  for  giving  me  more  than  enough 
rope  to  hang  myself.  Lastly,  I  thank  Mrs.  Asya  Campbell,  Mr.  Andy  Walther,  Mr.  Carlos 
Rexach,  of  the  Aerospace  Corporation,  and  Capt  Michael  Manning  for  their  instruction, 
advice  and  friendship.  I  believe  the  experiences  I  had  at  SMC  helped  me  greatly  during 
my  time  at  AFIT.  Those  mentioned  above,  as  well  as  many  unmentioned,  had  a  great 
influence  in  shaping  those  experiences. 


Robinson  Hughes 


Table  of  Contents 


Page 

Abstract . iv 

Dedication . v 

Acknowledgements . vi 

Table  of  Contents . vii 

List  of  Figures . ix 

List  of  Tables . x 

List  of  Acronyms . xi 

I.  Introduction . 1 

Some  Issues  with  DoD  Product  Development . 1 

The  Downward  Spiral . 3 

Genesis  of  a  Solution . 5 

Research  Questions . 7 

Method . 7 

Assumptions  and  Limitations . 8 

Significance  of  Study . 9 

Overview  of  Remaining  Chapters . 10 

II.  The  Product  Development  Literature . 11 

The  Development  Process . 11 

Influencing  the  Outcome . 14 

Definition  of  a  Product  Concept . 16 

Understanding  Need . 20 

Developing  the  Product  Concept . 22 

Resource  Allocation . 23 

DoD  Product  Development  Process . 25 

Proposed  Definition  of  a  DoD  Product  Concept . 28 

Measuring  Concepts . 29 

III.  Process  for  Developing  a  Framework  for  Material  Concept  Evaluation . 3 1 

A  Four-Step  process  to  develop  a  framework . 31 

Review  of  the  Product  Development  Literature . 3 1 

A  Definition  of  a  Material  Concept . 32 

-  vii  - 


Page 

The  Concept  within  the  Stage-Gate  Process . 33 

Detennine  the  Information  Required  for  a  Gate . 34 

IV .  F ramework  for  Evaluation . 36 

Context  of  the  Framework . 36 

Assessing  Maturity . 39 

Concept  Evaluation  and  Selection  within  a  Stage-Gated  Process . 40 

Gate  1 :  Opportunity  Identification  -  ICD  Approval  (AFROC/JROC) . 43 

Infonnation  Elements  for  Gate  1 . 44 

How  Much  is  Enough? . 47 

Gate  2:  Concept  Screening  -  MDD  (MDA) . 47 

Infonnation  Elements  for  Gate  2 . 48 

Gate  3:  Concept  Selection  -  Milestone  A  (MDA) . 51 

Infonnation  Elements  for  Gate  3 . 52 

An  Issue  of  Flexibility . 53 

The  Concept  Development  Process . 56 

V.  Conclusion,  Discussion  and  Recommendations . 58 

Overview  of  the  Research . 58 

Results  of  the  Research . 58 

Limitations  of  the  Research . 61 

Future  Research . 63 

Final  Words  and  Takeaways . 64 

Bibliography . 66 


-  viii  - 


List  of  Figures 


Page 

Figure  1  -  “Do  It  Right,  Do  It  Early;  Do  It  Early,  Do  It  Right”,  (Loren  &  Bullard, 

2008) .  16 

Figure  2  -  Comparison  of  Development  Processes .  25 

Figure  3  -  Information  Elements  for  Gate  1 . 45 

Figure  4  -  Information  Elements  for  Gate  2 .  49 

Figure  5  -  Information  Elements  for  Gate  3 .  53 

Figure  6  -  Concept  Maturity  Framework .  55 


-  ix  - 


List  of  Tables 


Page 

Table  1  -  DoDAF  Views  (  DoD  Deputy  Chief  Information  Officer) . 38 

Table  2  -  Architecture  Views  by  Gate .  57 


-  x  - 


List  of  Acronyms 


AFROC 

Air  Force  Requirements  Oversight  Council 

AoA 

Analysis  of  Alternatives 

AV 

All  View 

CV 

Capabilities  View 

CBA 

Capabilities  Based  Analysis 

CDI 

Child  Development  Inventory 

CJCS 

Chairman  of  the  Joint  Chiefs  of  Staff 

CJCSI 

Chairman  of  the  Joint  Chiefs  of  Staff  Instruction 

CJCSM 

Chairman  of  the  Joint  Chiefs  of  Staff  Manual 

CONOPS 

Concept  of  Operations 

DAG 

Defense  Acquisition  Guidebook 

DoD 

Department  of  Defense 

DoDAF 

Department  of  Defense  Architecture  Framework 

DoDI 

Department  of  Defense  Instruction 

GAO 

Government  Accounting  Office 

ICD 

Initial  Capabilities  Document 

JCIDS 

Joint  Capabilities  Integration  and  Development  System 

JROC 

Joint  Requirements  Oversight  Council 

LCC 

Life  Cycle  Cost 

MDA 

Milestone  Decision  Authority 

MDD 

Material  Development  Decision 

MOE 

Measure  of  Effectiveness 

MSA 

Material  Solution  Analysis 

MSA 

Milestone  A 

NPV 

Net  Present  Value 

NRC 

National  Research  Council 

OV 

Operational  View 

PPBE 

Planning,  Programming,  Budgeting  and  Execution 

ROI 

Return  on  Investment 

SAF/AQ 

Assistant  Secretary  of  the  Air  Force  for  Acquisitions 

SEP 

Systems  Engineering  Plan 

StdV 

Standards  View 

SV 

Systems  View 

SvcV 

Services  View 

-  XI  - 


DEVELOPMENT  OF  A  CONCEPT  MATURITY  ASSESSMENT  FRAMEWORK 

I.  Introduction 

The  Chairman  of  the  Joint  Chiefs  of  Staff  Instruction  (CJCSI)  3170.01G  estab¬ 
lishes  the  policies  and  the  procedures  for  the  Joint  Requirements  Oversight  Council 
(JROC)  to  identify  and  assess  joint  military  capability  needs  within  the  Joint  Capabilities 
Integration  and  Development  System  (JCIDS).  A  product  of  the  JCIDS  process  is  a  de¬ 
scription  of  where  the  capability  is  deficient  and  includes  a  recommended  solution  to  al¬ 
leviate  the  deficiency.  One  purpose  of  the  defense  acquisition  process  is  to  develop  and 
propose  system  concepts  as  possible  solutions  to  meet  the  capability  need.  CJCSI 
3 170.0 1G  discusses  types  of  infonnation  required  in  a  proposed  concept  but  does  not 
give  criteria  that  would  differentiate  between  a  well-developed  concept  that  is  ready  for 
consideration  and  one  that  is  immature.  This  thesis  presents  a  possible  solution:  a  scale 
of  maturity  levels  with  corresponding  criteria  that  the  decision  maker  can  use  to  evaluate 
concepts  prior  to  concept  selection  and  a  developer  can  use  to  build  and  mature  a  con¬ 
cept. 

Some  Issues  with  DoD  Product  Development 

Within  the  commercial  enterprise,  the  result  of  a  successful  development  project 
is  a  product  that  will  earn  profit  for  the  company  (Ulrich  &  Eppinger,  Product  Design  and 
Development,  2004).  The  government  does  not  earn  profit  from  the  systems  it  develops. 
According  to  Kaminski  and  Lyles  (2008)  the  results  of  a  successful  government  devel¬ 
opment  project  are:  the  system  meets  the  needs  of  the  users;  that  the  developers  deliver 
the  system  within  a  reasonable  amount  of  time;  and  that  the  cost  of  development  is  close 
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to  the  original  estimate.  The  Department  of  Defense  (DoD)  spends  billions  of  dollars 
each  year  to  develop  these  systems  but,  according  to  the  Government  Accountability  Of¬ 
fice  (GAO),  does  not  seem  to  be  getting  an  acceptable  return  on  investment.  Many  of  the 
major  systems  developed  by  the  Department  of  Defense  require  a  greater  amount  of  re¬ 
sources  than  was  originally  estimated  to  meet  the  user  needs  and  systems  are  often  deli¬ 
vered  later  than  originally  estimated,  sometimes  by  years  (GAO,  Best  Practices: 

Capturing  design  and  manufacturing  knowledge  early  imroves  acquisition  outcomes  - 
GAO-02-701,  2002). 

There  are  many  suggested  reasons  to  explain  why  the  programs  of  late  have  not 
met  their  expected  outcomes.  The  Government  Accountability  Office,  the  investigative 
organization  of  the  U.S.  Congress,  cites  issues  that  have  led  to  the  increase  in  cost  and 
time  of  programs,  which  include  unstable  requirements,  the  use  of  immature  technology, 
and  the  failure  to  match  required  resources  to  programs  (GAO,  Defense  Acquisition: 
Employing  best  practices  can  shape  better  weapon  systems  -  GAO/T-NSIAD-OO-137, 
2000).  Other  authors  like  Suddarth  (2002)  have  argued  the  reason  for  relatively  poor  per¬ 
formance  of  recent  program  development  is  that  the  systems  are  more  complex  and  the 
methods  of  developing  systems  have  diminished  over  the  years.  In  a  study  conducted  in 
2008  by  the  National  Academy  of  Sciences,  a  committee  chaired  by  Dr.  Paul  Kaminski 
and  Gen  (ret.)  Lester  Lyles  (USAL)  identified  just  about  all  of  the  suggested  reasons  for 
poor  performance  stated  above  and  noted  that  the  root  of  the  problem  is  not  simple.  The 
committee  noted  that  the  causes  of  poor  program  performance  and  their  effects  are  “com¬ 
plex  and  interrelated”,  which  makes  establishing  causality  by  quantitative  means  nearly 
impossible  (Kaminski  &  Lyles,  2008,  p.  10).  However,  the  committee  did  believe  that 
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early  systems  engineering  does  contribute  to  successful  program  outcomes  and  found  in 
its  research  that  the  only  DoD  programs  that  that  were  successful  had  instituted  systems 
engineering  before  concept  selection.  The  committee  did  qualify  that  early  application  of 
systems  engineering  is  necessary  for  a  successful  development  project  but  it  is  not  suffi¬ 
cient  to  ensure  a  successful  outcome.  The  committee  did  find  several  programs  that  insti¬ 
tuted  systems  engineering  early  in  the  development  cycle  and  were  not  successful,  but 
every  program  that  ignored  early  systems  engineering  failed  (Kaminski  &  Lyles,  2008). 

Within  the  DoD  acquisition  process,  the  development  of  the  “alternatives  for  the 
solution”  (CJCS,  2009,  pp.  A-5)  is  where  the  use  of  early  systems  engineering  could  be 
applied.  These  proposed  solutions  are  in  the  fonn  of  a  product  or  system  concept.  The 
concepts  are  intended  to  provide  a  solution  to  a  deficient  military  capability  and  identify 
the  resources  required  to  develop  the  solution  (DoD,  DoD  Instruction  5000.02,  2008). 

The  similarities  of  the  DoD  concept  to  those  found  in  the  commercial  equivalent  will  be 
explored  in  Chapter  2.  However,  for  the  purpose  of  this  research,  the  working  definition 
of  a  DoD  concept  is  a  solution  to  address  an  identified  need  and  the  identification  of  the 
required  resources  to  develop  that  solution.  The  solution  described  by  the  concept  can  be 
a  single  product,  a  system  or  a  system  of  systems  depending  upon  the  deficient  military 
capability. 

The  Downward  Spiral. 

Before  the  Department  of  Defense  begins  a  program  it  is  required  to  develop  es¬ 
timates  of  when  a  program  will  be  complete,  what  resources  are  required  and  how  much 
it  will  cost  to  develop,  deploy,  and  operate  the  system  (DoD,  2008).  The  government 
cost  analysts  require  information  gathered  early  in  the  development  cycle  to  develop 
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these  estimates  but  often  rely  heavily  on  assumptions  to  develop  the  cost  estimates  due  to 
a  dearth  of  information  (GAO,  A  Knowledge  Based  Funding  Approach  Could  Improve 
Major  Weapon  System  Program  Ouotcomes  GAO-08-619,  2008).  Because  these  esti¬ 
mates  have  proven  less  than  accurate  the  government  starts  development  programs  with¬ 
out  correctly  matching  the  needed  resources  to  the  development  program  (GAO,  Best 
Practices:  Better  matching  of  needs  and  resources  will  lead  to  better  weapon  system 
outcomes  -  GAO-Ol-288,  2001).  Often  when  a  program  manager  should  be  focusing  on 
design  and  development,  he  is  unexpectedly  forced  to  apply  resources  to  issues  like  ma¬ 
turing  technology  and  clarifying  requirements  that  could  have  been  addressed  prior  to  the 
start  of  a  program  (GAO,  Defense  Acquisition:  Employing  best  practices  can  shape  better 
weapon  systems  -  GAO/T-NSIAD-OO-137,  2000).  According  to  Repenning,  Gonsalves, 
and  Black  (2001)  the  process  of  allocating  resources  to  unanticipated  problems,  called 
firefighting,  can  become  the  de  facto  process  in  a  development  program  if  the  earlier 
phases  of  a  project  are  not  given  adequate  attention.  Even  a  temporary  increase  in  work 
due  to  unanticipated  problems  that  results  in  firefighting  can  permanently  degrade  a  de¬ 
velopment  team’s  performance.  Once  introduced  into  an  organization,  firefighting  can 
become  a  very  expensive  self-reinforcing  plague  that  can  quickly  spread  through  an  en¬ 
tire  development  system.  The  effect  of  all  these  issues  is  that  performance  capabilities  of 
the  systems  are  over-promised  and  the  cost  of  the  system,  to  build  and  maintain,  are  un¬ 
der-estimated. 

A  second  order  effect  further  exacerbates  the  problem  within  government  devel¬ 
opment  programs.  Government  Accounting  Office  reports  (GAO-Ol-288  (2001)  and 
GAO/T-NSSIAD-OO-137  (2000))  indicate  that  when  the  government  fields  development 
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programs  and  the  true  cost  to  operate  a  system  is  greater  than  was  budgeted,  the  money  to 
allow  the  continued  operation  of  the  fielded  systems  comes  from  programs  still  in  devel¬ 
opment.  The  withdrawal  of  funds  from  the  developing  programs  forces  delays  in  the  de¬ 
velopment,  increases  the  cost  of  further  development,  or  causes  the  acquisition  leadership 
to  cancel  the  program.  Either  of  these  actions  delays  the  arrival  of  the  needed  capability. 
The  final  result  is  the  user  receives  a  system  that  does  not  perform  as  expected  and  costs 
more  than  anticipated  (GAO,  Defense  Acquisition:  Employing  best  practices  can  shape 
better  weapon  systems  -  GAO/T-NSIAD-OO-137,  2000). 

It  is  not  difficult  to  connect  the  fiscal  and  temporal  effects  of  poor  program  plan¬ 
ning  when  one  understands  that  decisions  that  determine  50-80%  of  a  system’s  entire 
Life  Cycle  Cost  are  made  before  a  solution  is  chosen  at  Milestone  A  (Kaminski  &  Lyles, 
2008  &  LT  Gen  Shakleford,  2009  &  Blanchard  &  Fabrycky,  2006).  The  issues  expe¬ 
rienced  in  product  development  are  challenging  but  can  be  addressed  by  the  application 
of  proper  management  practices  early  in  the  research  and  development  cycle,  where 
management  has  the  greatest  influence  on  a  project’s  outcome  (Wheelwright  &  Clark, 
1992). 

Genesis  of  a  Solution. 

Some  management  practices  for  the  early  development  of  complex  systems  are 
found  in  the  discipline  of  systems  engineering  (GAO,  A  Knowledge  Based  Funding 
Approach  Could  Improve  Major  Weapon  System  Program  Ouotcomes  GAO-08-619, 
2008).  According  to  Blanchard  and  Fabrycky  (2006),  there  is  no  one  accepted  definition 
of  systems  engineering.  However,  the  objective  of  systems  engineering  is  quite  clear.  It 
is  to  translate  needs  into  a  solution.  The  discipline  of  systems  engineering  involves  an 
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approach  that  views  the  whole  system  and  adopts  a  life-cycle  orientation,  which  ad¬ 
dresses  all  phases  of  a  system’s  life,  from  the  identification  of  a  need  to  the  disposal  of 
the  system.  Because  the  discipline  of  systems  engineering  is  applied  at  the  early  part  of 
the  development  process,  it  utilizes  non-quantitative  analysis  methods  of  systems  archi¬ 
tecting  (Maier  &  Rechtin,  2002).  The  system  wide  focus  and  disciplined  approach  force 
a  development  team  to  ask  questions  very  early  in  the  development  process  that  greatly 
influence  a  system’s  outcome.  The  work  conducted  early  has  three  benefits  for  the  de¬ 
veloper:  a  reduction  in  the  life  cycle  cost,  reduced  acquisition  time,  and  greater  visibility 
into  the  system,  which  should  reduce  the  potential  risks  (Blanchard  &  Fabrycky,  2006). 
Blanchard  and  Fabrycky  (2006)  state  that  an  additional  benefit  of  the  visibility  that  comes 
from  applying  systems  engineering  is  an  understanding  of  what  types  of  resources  a  de¬ 
veloper  will  need  throughout  the  development  process. 

At  present,  the  guiding  and  instructing  documents  for  Air  Force  early  system  de¬ 
velopment  indicate  types  of  information  that  decision  makers  require  but  do  not  indicate 
or  suggest  the  fidelity  of  the  infonnation  (Assistant  Secretary  of  the  Air  Force  for  Acqui¬ 
sition,  2009  and  CJCS,  2007)).  This  lack  of  definition  applies  to  all  aspects  of  early 
product  development,  including  the  system  concepts  proposed  to  meet  a  capability  short¬ 
fall.  This  lack  of  definition  of  what  a  complete  concept  should  be  adds  an  additional  de¬ 
gree  of  difficulty  to  the  task  of  choosing  a  concept  for  development.  Those  responsible 
for  selecting  a  solution  to  a  capability  shortfall  must  detennine  the  degree  of  fidelity  and 
completeness  of  the  information  upon  which  they  are  about  to  make  a  decision  without 
the  benefit  of  a  reference  with  which  to  compare.  Therefore,  this  research  seeks  to  de¬ 
velop  a  framework  with  which  to  evaluate  the  completeness  of  a  concept. 
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Research  Questions 

This  research  will  identify  aspects  of  a  concept  that  are  vital  to  create  a  solid  plan 
for  a  successful  development  program.  By  analyzing  the  best  practices  of  commercial 
enterprises  and  applying  the  principles  of  systems  architecture  and  systems  engineering, 
this  work  will  recommend  levels  of  maturity  that  will  give  the  decision  maker  and  the 
developers  an  indication  of  how  complete  a  concept  is.  This  research  will  attempt  to  an¬ 
swer  the  following  questions: 

1 .  What  is  the  definition  of  a  product  concept  as  it  applies  to  DoD  capa¬ 
bility  development? 

2.  What  type  of  information  does  a  concept  require  to  be  mature  in  a 
stage-gated  process? 

3.  How  can  the  definition  of  a  mature  concept  account  for  the  phases  of 
development? 

4.  What  is  the  infonnation  (criteria)  required  at  each  gate  in  the  process? 

5.  What  potential  architecture  views  capture  the  information  required  by 
decision  makers  at  each  gate? 

6.  What  information,  if  any,  is  important  to  the  maturity  of  a  concept  but 
are  not  required  by  JCIDS  or  the  DoDI  5000  series? 

Method 

First,  a  review  of  the  literature  addressing  commercial  product  development,  De¬ 
partment  of  Defense  acquisition  policy  and  guidance,  Air  Force  policy  and  guidance  on 
acquisition  processes,  and  systems  engineering  and  architecting  was  completed.  Addi¬ 
tionally,  an  search  was  conducted  on  the  existing  literature  concerning  product  concepts 
and  their  maturity. 
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The  existing  literature  on  product  development  was  used  in  conjunction  with  the 
policy  and  guidance  of  DoD  acquisition  to  detennine  the  relevant  decision  gates  when  a 
concept  would  be  evaluated.  Then,  with  the  goal  of  having  a  complete  concept  at  the 
concept  selection  gate,  e.g.  Milestone  A,  the  type  and  amount  of  information  required  at 
each  decision  gate  was  identified.  The  detailed  process  used  is  described  in  Chapter  3. 
Assumptions  and  Limitations 

This  research  is  based  on  the  best  practices  of  industry  and  assumes  that  the  prin¬ 
ciples  of  product  development  found  in  industry  can  also  be  applied  to  product  develop¬ 
ment  within  the  Department  of  Defense.  While  the  environment  and  motivations  of  the 
two  communities  may  differ,  the  effects  of  actions  conducted  early  in  the  product  devel¬ 
opment  process  should  not.  The  processes  of  each  domain  are  designed  to  apply  re¬ 
sources  to  the  development  of  an  idea  in  order  to  create  a  product  that  meets  an  identified 
need. 

This  research  is  not  intended  to  recommend  changes  to  the  current  Acquisition 
process  detailed  in  the  JCIDS  and  DoD  policy.  This  research  was  conducted  under  the 
assumption  that  any  outcome  or  recommendation  would  be  implemented  within  the  con¬ 
text  of  the  Department  of  Defense  acquisition  process,  including  its  development  phases 
and  decision  points.  However,  this  research  will  identify  where  and  when  data  should  be 
included  in  the  concept  that  may  differ  from  what  is  found  in  existing  guidance. 

However,  the  assumption  that  the  product  of  this  research  will  be  used  in  the  DoD 
acquisition  process  leads  to  a  flawed  assessment  of  its  potential.  The  primary  limitation 
of  this  research  is  that  it  focuses  primarily  on  how  to  gauge  the  completeness  of  a  con¬ 
cept.  The  purpose  of  the  assessment,  using  the  proposed  framework,  is  only  to  address 
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the  completeness  and  fidelity  of  the  concept,  not  to  address  how  well  the  concept  meets 
the  needs  of  the  intended  user.  The  assessment  will  identify  the  rigor  or  sufficiency  of 
any  analyses  by  detennining  if  the  right  type  and  amount  of  infonnation  is  available.  Be¬ 
cause  the  product  development  process  contains  a  common  goal  and  similar  framework, 
the  products  of  this  research  could  be  applied  to  a  context  other  than  the  DoD.  Addition¬ 
al  research  into  further  application  of  the  products  would  be  required. 

Significance  of  Study 

The  literature  review  documented  in  Chapter  2  of  this  report  shows  that  there  is 
information  available  about  product  concepts  and  the  importance  of  product  concepts  in 
the  development  process  but  there  is  little  infonnation  defining  what  a  mature  or  com¬ 
plete  concept  is.  However,  there  is  interest  within  the  Air  Force  and  other  federal  gov¬ 
ernment  agencies  to  develop  an  ability  to  evaluate  the  completeness  of  concepts  (Vane, 
2009).  This  research  creates  a  framework  for  the  development  and  assessment  of  product 
concepts.  The  framework  can  be  used  by  organizations  to  guide  the  development  of  a 
product  concept,  and  decision  makers  can  use  it  to  evaluate  a  proposed  product  concept. 
The  framework  should  be  used  to  mature  a  concept  so  the  decision  maker  has  sufficient 
information  to  judge  the  suitability  and  effectiveness  of  each  concept  and  detennine 
which  concept  (if  any)  should  continue  to  the  next  phase  of  development.  The  informa¬ 
tion  within  the  framework  is  not  intended  to  be  used  as  the  criteria  in  a  “check  the  box” 
event  that  allows  a  concept  to  proceed  through  a  gate  just  because  it  is  mature. 
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Overview  of  Remaining  Chapters 

The  remaining  chapters  introduce  the  concepts  necessary  to  understand  this  re¬ 
search,  review  the  methodology  used  and  the  subsequent  results,  and  draw  conclusions 
and  recommendations.  Chapter  2  provides  a  review  of  the  existing  literature  about  prod¬ 
uct  development.  It  will  provide  a  comparison  of  the  government  and  commercial  prod¬ 
uct  development  processes,  review  the  importance  of  beginning  the  process  with  a  com¬ 
plete  concept  and  reviews  a  working  definition  of  DoD  product  concept.  Methodology  is 
discussed  in  Chapter  3.  Chapter  4  describes  the  framework  that  was  created  to  develop 
and  assess  a  DoD  product  concept  within  the  early  phases  of  the  DoD  stage-gated 
process.  Finally,  Chapter  5  interprets  the  results  of  the  analysis,  draws  conclusions,  and 
makes  recommendations  for  further  research. 
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II.  The  Product  Development  Literature 


This  chapter  will  examine  some  of  the  thinking  about  the  purpose  and 
function  of  a  product  development  process  in  commercial  industry.  It  will  then  examine 
the  purpose  of  the  front-end  of  product  development  and  the  influence  it  has  on  the  whole 
process. 

The  Development  Process 

Organizations  whose  purpose  is  to  develop  solutions,  be  they  in  the  fonn 
of  physical  systems  or  processes,  struggle  with  the  challenges  of  designing  a  develop¬ 
ment  process  that  consistently  produces  effective  solutions  to  meet  the  needs  of  their  cus¬ 
tomers.  According  to  Hammer  and  Champy  (2003),  the  difference  between  companies 
that  win  and  those  that  lose  is  “that  winning  companies  know  how  to  do  their  work  bet¬ 
ter”  (pg  29).  The  companies  that  have  designed  their  processes  to  meet  the  needs  of  their 
customers  with  profitable  solutions  succeed  in  business.  Those  companies  that  design  a 
process,  which  either  fails  to  identify  the  customer’s  need  or  fails  to  create  an  acceptable 
solution  to  an  identified  need,  do  not  succeed. 

Wheelwright  and  Clark  (1992)  contend  that  the  goal  of  any  development 
project  is  to  “take  an  idea  from  concept  to  reality”  by  developing  a  product  that  meets  the 
identified  need  and  is  manufacturable  (pg.  111).  They  use  the  idea  of  a  funnel  to  describe 
the  process  to  identify  and  select  products  for  development.  The  funnel  is  composed  of 
phases  separated  by  decision  points  and  is  designed  to  force  a  company  to  gather  con¬ 
cepts,  improve  the  concepts,  select  only  the  best  ones  to  develop,  allocate  resources  to  the 
project  and  turn  the  concepts  into  products.  At  each  phase  of  the  process,  enough  infor¬ 
mation  is  collected  to  allow  the  decision  makers  to  correctly  assess  a  project  and  deter- 
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mine  if  additional  resources  should  be  allocated  to  its  development.  The  process  is  de¬ 
signed  to  identify  many  potential  solutions  but  to  select  for  development  only  the  ones 
that  meet  the  identified  need  and  are  complete  (Wheelwright  &  Clark,  1992). 

Ulrich  and  Eppinger  (2004)  describe  the  generic  product  development  process  in 
terms  of  six  phases:  Planning,  concept  development,  system  level  design,  detail  design, 
testing  and  refinement,  and  production  ramp  up.  The  generic  process  is  intended  to  be 
tailored  to  the  specific  developmental  context  in  order  to  help  the  organization  achieve 
the  goal  of  identifying  customer  needs  and  developing  a  marketable  solution  to  meet 
those  needs.  Ulrich  and  Eppinger  (2004)  define  a  process  as  “a  sequence  of  steps  that 
transforms  a  set  of  inputs  into  a  set  of  outputs”  (pg  12).  From  their  definition,  each  phase 
of  the  entire  process  can  be  considered  a  process  as  the  output  of  a  phase  is  the  necessary 
input  to  the  following  phase.  If  the  transition  from  one  phase  to  another  is  used  as  a 
check-point  to  prevent  immature  projects  from  progressing  to  additional  phases,  the  de¬ 
velopment  process  can  help  assure  a  quality  product  (Ulrich  &  Eppinger,  Product  Design 
and  Development,  2004). 

Ulrich  and  Eppinger  (2004)  further  suggest  that  the  product  development  process 
can  be  looked  at  in  three  ways.  First,  the  process  can  be  viewed  as  a  system  to  devel¬ 
op/select  a  concept  and  transfonn  it  into  a  product.  Second,  the  development  process  can 
be  viewed  as  an  information-processing  system,  which  creates  concepts,  designs  and  spe¬ 
cifications  based  upon  the  needs  of  the  users,  the  goals  of  the  organization  and  the  availa¬ 
ble  resources.  Finally,  the  process  can  be  viewed  as  a  risk  management  system,  which 
identifies  and  prioritizes  risks  early  in  the  development  process  and  allows  the  develop¬ 
ment  team  to  eliminate  key  uncertainties  and  reduce  risk. 
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Cooper’s  (1990)  description  of  the  development  process  is  similar  to  those  of 
Wheelwright  &  Clark  (1992)  and  Ulrich  and  Eppinger  (2004).  His  process  begins  with 
an  assessment  of  the  ideas  that  are  intended  to  meet  market  needs  and  ends  with  the  in¬ 
troduction  of  the  product  into  the  market.  Cooper  places  great  emphasis  on  designing  a 
complete  process  and  using  the  “gates”  between  stages  to  exercise  a  “go/kill/hold/recycle 
decision”  (pg46)  so  that  no  activities  critical  to  the  success  of  the  project  are  omitted. 
Cooper  argues  the  gates  must  be  used  as  quality  control  checkpoints,  each  with  a  set  of 
criteria,  and  that  no  project  be  allowed  to  proceed  to  the  next  phase  of  development  with¬ 
out  meeting  the  criteria.  The  idea  of  using  a  complete  process  made  up  of  phases  and 
gates  for  product  development  is  consistent  with  the  recommendations  of  Wheelwright  & 
Clark  (1993),  Ulrich  and  Eppinger  (2004),  and  with  the  practice  of  the  Department  of  De¬ 
fense  (DoD,  DoD  Instruction  5000.02,  2008). 

Krishnan  and  Ulrich  (2001)  define  product  development  as  “the  transformation  of 
a  market  opportunity  and  set  of  assumptions  about  product  technology  into  a  product 
available  for  sale”  (pg  1).  The  definition  is  a  bit  more  general  than  the  processes  de¬ 
scribed  earlier  but  captures  the  same  intent  of  the  more  detailed  processes.  Krishnan  and 
Ulrich  (2001)  looked  at  the  decisions  that  are  made  during  development  and  contend  that 
how  organizations  develop  items  may  differ  greatly  from  one  to  another,  but  what  is  be¬ 
ing  decided  is  consistent  at  a  certain  level  of  abstraction.  Each  organization  makes  deci¬ 
sions  by  design  or  default  on  concepts,  architecture,  configuration,  logistics,  and  project 
schedule. 

Each  of  the  processes  described  here  are  designed  to  help  ensure  a  successful  out¬ 
come  of  a  development  project.  However,  knowing  what  a  well  designed  process  should 
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contain  does  not  ensure  success.  Even  with  the  availability  of  a  vast  amount  of  literature 
on  product  development,  creating  a  successful  development  process  is  challenging. 

Some  research  has  shown  that  practice  does  not  always  follow  theory  (Repenning, 
Gonsalves,  &  Black,  2001).  Cooper  and  Kleinschmidt  (1986)  conducted  a  study  of  203 
new  product  projects  and  found  that  many  of  the  projects  encountered  difficulties  because 
processes  were  incomplete  or  were  executed  poorly.  In  reviewing  this  data,  Cooper 
(1990)  found  that  there  was  a  strong  link  between  quality  execution  of  the  process  and  a 
successful  outcome.  Cooper  (1990)  also  found  that  the  activities  that  had  the  greatest 
impact  on  the  success  of  a  product  were  found  in  the  early  stages  of  development.  The 
collection  of  activities  that  precede  the  design  and  production  of  a  product  are  often  re¬ 
ferred  to  as  the  “fuzzy  front-end.”  It  is  in  this  phase  that  the  opportunities  are  found, 
ideas  are  created,  and  concepts  are  selected  (Dahan  &  Hauser,  2001). 

Influencing  the  Outcome 

Within  the  product  development  literature  there  has  been  a  great  emphasis  on  un¬ 
derstanding  the  factors  that  determine  the  success  or  failure  of  a  development  project. 
Several  authors  have  attempted  to  synthesize  the  growing  body  of  literature  about  success 
factors  to  gain  a  better  understanding  of  product  development.  Brown  and  Eisenhardt 
(1995)  highlighted  the  “the  distinction  of  process  perfonnance  and  product  effectiveness 
and  the  importance  of  agents”  acting  on  that  process  (pg  343).  The  developing  organiza¬ 
tion  may  have  little  or  no  control  upon  the  influence  that  agents  have  on  the  development 
process,  especially  those  agents  outside  the  boundaries  of  the  organization.  However,  the 
development  process  itself  and  the  execution  of  the  process  are  within  the  control  of  the 
development  organization  (Balachandra  &  Friar,  1997;  Krishnan  &  Ulrich,  2001). 
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Cooper  (1990)  contends  that  most  products  fail  due  to  “errors  of  omission  and 
commission”  (pg  48)  in  the  process.  Some  of  these  errors  surface  in  the  form  of  not  un¬ 
derstanding  user  needs,  defects  in  the  product,  and  poor  project  evaluation  and  screening. 
Cooper  (1990)  found  that  the  activities  that  precede  the  product  development  phase  had 
the  greatest  impact  on  a  projects  success  because  they  “qualify  and  define”  (pg49)  the 
project.  The  projects  that  executed  rigorous  front-end  activities,  or  did  their  “homework” 
(pg  49),  had  the  greatest  chance  for  success. 

Khuranan  and  Rosenthal  (1997)  argue  that  to  execute  the  front-end  activities  cor¬ 
rectly  an  organization  should  treat  the  activities  of  need  identification,  project  planning 
and  concept  generation  as  interrelated  parts  of  a  process  rather  than  independent  activi¬ 
ties.  They  separated  activities  within  an  organization’s  front-end  process  into  founda¬ 
tional  activities  that  span  across  the  organization’s  development  portfolio  and  project- 
specific  activities.  The  purpose  of  the  project  specific  activities  is  to  “clarify  the  product 
concept,  define  product  and  [user]  requirements,  and  develop  plans,  schedules,  and  esti¬ 
mates  of  the  project’s  resource  requirements”  (pg  104).  Though  some  research  indicates 
that  the  many  factors  and  their  relative  impact  to  the  success  of  a  project  are  contextual 
(Balachandra  &  Friar,  1997),  the  preponderance  of  the  literature  indicate  the  activities  of 
understanding  user  needs,  developing  the  product  concept,  and  allocating  sufficient  re¬ 
sources  have  a  great  impact  on  the  subsequent  development  phases.  The  focus  placed  on 
these  activities  is  understandable  since  the  decisions  that  determine  50-80%  of  a  system’s 
entire  Life  Cycle  Cost  (Figure  1)  are  made  before  a  solution  is  chosen  (Kaminski  & 
Lyles,  2008  &  LT  Gen  Shakleford,  2009  &  Blanchard  &  Fabrycky,  2006). 
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Development  Integration  Verification  Fielding  Operation 


Figure  1  -  “Do  It  Right,  Do  It  Early;  Do  It  Early,  Do  It  Right”,  (Loren  &  Bullard,  2008) 


Definition  of  a  Product  Concept. 

To  understand  why  a  product  concept,  also  referred  to  as  a  concept,  has  such  in¬ 
fluence  on  the  outcome  of  a  project  it  is  necessary  to  understand  what  is  in  a  concept. 
Here,  like  the  success  factors,  the  literature  varies.  While  a  few  researchers  offer  an  ex¬ 
plicit  definition  of  what  a  concept  is,  most  describe  the  actions  taken  during  concept  de¬ 
velopment,  the  inputs  to  a  concept,  the  outputs  of  a  concept,  and/or  the  purpose  of  a  con¬ 
cept  and  of  concept  development.  However,  from  these  descriptions,  a  good  understand¬ 
ing  of  what  a  concept  “is”  can  be  developed. 

Ulrich  and  Eppinger  (2004)  define  a  concept  as  “a  description  of  the  form,  func¬ 
tion,  and  features  of  a  product”  (pg  15)  and  that  it  is  a  description  of  “how  the  product 
will  satisfy  the  customer  needs”  (pg  98).  Some  of  the  activities  that  an  organization  ex- 
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ecutes  to  develop  a  concept  include  identifying  customer  needs;  developing  target  speci¬ 
fications;  generating,  selecting  and  testing  a  concept;  setting  final  product  specifications; 
and  planning  the  project,  which  includes  determining  resource  requirements  to  develop  a 
schedule  and  budget.  Wheelwright  and  Clark  (1992)  describe  the  concept  development 
phase  as  one  where  infonnation  is  gathered  on  a  market  opportunity  and  ideas  are  devel¬ 
oped  for  that  opportunity.  The  ideas  are  then  tested  to  see  if  they  meet  the  needs  of  the 
market  and  if  the  development  of  the  concept  will  be  economically  beneficial  to  the  com¬ 
pany.  Bacon  et  al  (1994)  have  a  slightly  different  name  for  what  Ulrich  and  Eppinger 
(2004)  and  Wheelwright  and  Clark  (1992)  call  the  concept,  the  product  definition.  The 
activities  executed  to  develop  the  product  definition  are  similar  to  those  executed  for 
concept  generation:  assessment  of  customer  and  user  needs,  identification  of  technical 
risks  and  opportunities,  and  assessment  of  the  market  environment  in  which  the  product 
will  be  offered.  The  outputs  of  the  product  definition  process  are  descriptions  of  func¬ 
tions,  features  and  price  of  the  product;  an  understanding  of  the  required  technologies; 
and  allocation  of  enough  resources  to  complete  development.  Cooper  (1993)  considers 
the  product  concept  as  part  of  the  product  definition.  His  description  of  what  information 
is  required  to  be  in  the  product  definition  closely  matches  what  Ulrich  and  Eppinger 
(2004)  suggest  needs  to  be  in  the  product  concept.  Burchill’s  (1993)  process  to  develop  a 
concept  also  begins  with  activities  to  understand  the  user’s  need  and  environment.  Re¬ 
quirements  and  specifications  are  developed  from  the  needs,  solutions  are  created  during 
concept  development,  and  then  a  concept  is  selected  for  development.  The  concepts  are 
evaluated  against  customer  requirements  and  organizational  constraints  which  implies  the 
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resources  required  to  develop  the  product  are  measured  against  what  resources  the  organ¬ 
ization  has  available. 

Repenning  et  al  (2001)  state  “the  primary  role  of  concept  development  activities 
is  to  make  downstream  design  work  more  effective”  (pg.  48).  The  way  the  activities  of 
concept  generation  accomplish  this  is  by  understanding  and  documenting  user  needs 
(Repenning,  Gonsalves,  &  Black,  2001),  “exploiting]  the  space  of  product  concepts  that 
may  address  the  customer  needs”  to  find  the  best  concepts  (Ulrich  &  Eppinger,  Product 
Design  and  Development,  2004,  p.  16),  and  integrating  requirements,  created  from  user 
needs,  into  structured  design  activities  (Burchill,  1993).  Additional  objectives  that  should 
be  met  from  the  successful  development  of  a  product  concept  are:  linking  the  project  to 
the  organization’s  goals  (Wheelwright  &  Clark,  1992),  developing  product  architecture 
(Ulrich  &  Eppinger,  2004  and  Krishnan  &  Ulrich,  2001),  identifying  required  technology 
and  assessing  the  risk  associated  with  it  (Wheelwright  &  Clark,  1992  and  Bacon, 
Beckman,  Mowery,  &  Wilson,  1994),  identifying  project  risk  (Bacon,  Beckman, 

Mowery,  &  Wilson,  1994),  and  detennining  the  amount  and  type  of  resources  necessary 
to  complete  the  project  (Burchill,  1993  and  Bacon,  Beckman,  Mowery,  &  Wilson,  1994 
and  Wheelwright  &  Clark,  1992). 

The  evidence  for  accepting  these  objectives  listed  above  can  be  observed  by  look¬ 
ing  at  what  is  required  at  the  conclusion  of  Intel  Corporation’s  initial  two  phases  of  prod¬ 
uct  development.  A  cross-functional  team  develops  the  concept  and  product  definition  in 
the  first  phase  and  then  prepares  a  business  plan  for  an  approval  meeting.  The  business 
plan  contains  “a  return  on  investment/net  present  value  (ROI/NPV)  analysis,  target  cus¬ 
tomers,  high-level  product  requirements,  major  risks,  an  estimate  of  resources  needed,  a 
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preliminary  schedule”  as  well  as  an  estimate  of  the  feasibility  of  the  plan  given  its  tech¬ 
nical  and  timing  considerations  (Rafinejad,  2007,  p.  163).  The  approval  meeting  is  held 
to  determine  if  the  project  should  proceed  to  the  product  planning  phase  where  architec¬ 
ture,  design  and  plans  are  developed  to  “enable  due  diligence  planning”  (pg.  163)  and 
where  a  detailed  implementation  plan  is  developed.  At  the  end  of  this  second  phase  suf¬ 
ficient  architecture,  design  and  plans  have  been  created  to  describe  how  requirements  will 
be  realized,  and  a  development  plan  has  been  created  that  contains  a  detailed  program 
schedule  and  resource  plan  for  the  subsequent  phases,  a  risk  assessment,  and  identifies 
internal  and  external  dependencies.  Intel’s  management  must  then  decide  if  they  wish  to 
apply  resources  to  pursue  the  plan  as  it  was  proposed  by  the  development  team 
(Rafinejad,  2007). 

While  the  terms  and  the  definitions  of  the  terms  vary,  the  type  of  information  that 
is  required  in  the  product  concept  is  consistent.  Three  categories  of  information  emerge 
from  the  literature:  needs,  solutions,  and  resources.  The  literature  agrees  that  once  an  or¬ 
ganization  decides  to  pursue  a  development  opportunity,  the  next  action  that  should  be 
taken  is  to  identify  and  understand  the  needs  of  the  customers  and  users,  which  includes 
the  environment  in  which  the  product  will  be  used.  The  next  part  of  the  concept  genera¬ 
tion  process  is  to  find  a  solution  that  meets  the  needs  of  the  identified  users.  An  area  of 
concept  generation  where  the  authors  do  not  agree  is  the  development  of  product  specifi¬ 
cations.  Bacon  et  al.  (1994)  and  Khurana  and  Rosenthal  (1997)  suggest  that  technical 
product  specifications  need  not  be  developed  before  a  concept  is  selected;  Ulrich  and  Ep- 
pinger  (2004),  Cooper  (1993),  and  Bruchill  (1993)  argue  that  target  specifications  devel¬ 
oped  from  the  user  needs  are  essential  to  the  development,  assessment  and  selection  of  a 
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concept.  The  last  type  of  infonnation  found  in  the  product  concept  is  identification  of  the 
resources  required  to  develop  the  concept  into  a  product.  It  is  essential  that  companies 
ensure  they  understand  the  cost  of  development  and  the  anticipated  selling  price  to  de¬ 
termine  if  the  concept  is  worth  pursuing  (Cooper  R.  G.,  1990). 

Understanding  Need. 

Khurana  and  Rosenthal  (1997)  suggest  that  once  an  organization  identifies  an  op¬ 
portunity  the  first  step  in  the  development  process  is  a  preliminary  identification  of  cus¬ 
tomer  needs  and  a  survey  of  the  user  environment.  If  the  organization  deems  the  oppor¬ 
tunity  worth  pursuing  the  next  phase  begins  with  a  better  understanding  of  user  needs. 
Ulrich  &  Eppirnger  (2004),  Dahan  &  Hauser  (2001),  and  Wheelwright  and  Clark  (1992) 
suggest  engaging  in  activities  to  develop  a  greater  understanding  of  user  needs  is  the  ini¬ 
tial  step  in  the  development  process.  Similarly,  the  activities  to  identify  and  define  the 
requirements  for  DoD  development  programs  are  executed  in  a  Capabilities  Based  As¬ 
sessment  (CBA)  at  the  beginning  of  the  JCIDS  process.  The  purpose  of  the  CBA  is  to 
validate  any  gaps  in  capability  and  to  identify  the  mission,  the  operating  environment, 
and  any  operational  attributes  and  characteristics  associated  with  the  needed  capabilities 
(CJCS,  2009).  The  marketing  department  found  in  a  corporation  is  the  commercial  ana¬ 
logue  to  the  JCIDS  process.  The  role  of  marketing  is  to  “ensur[e]  that  the  company  is 
delivering  value”  by  defining  who  the  right  customers  are,  what  those  customers  need, 
and  what  will  meet  that  need  and  deliver  value  (Rafinejad,  2007,  p.  60).  The  reason  that 
each  of  these  processes  begin  with  the  identification  of  the  intended  user  needs  is  because 
ultimately  the  user  acts  as  the  final  judge  to  determine  the  success  of  the  project  and  the 
process  of  understanding  user  criteria  allows  an  organization  to  propose  solutions  the  us- 
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ers  will  find  favorable  (Shocker  &  Srinivasan,  1979).  Dahan  and  Hauser  (2001)  state 
clearly  “customers  do  not  buy  products  that  do  not  satisfy  their  needs”  (pg  25). 

Understanding  user  needs  well  in  advance  of  any  product  design  is  integral  to  the 
larger  product  development  process  (Ulrich  &  Eppinger,  Product  Design  and 
Development,  2004)  and  there  are  three  beneficial  reasons  to  execute  a  rigorous  activity 
to  accomplish  the  goal.  First,  these  early  activities  define  the  project  (Cooper  R.  G., 
1990).  Once  the  needs  are  captured  they  must  be  effectively  communicated  to  the  design 
team  so  that  the  design  team  can  translate  the  customer  needs  into  specifications  for  a 
technical  solution  (Ulrich  &  Eppinger,  Product  Design  and  Development,  2004).  Second, 
when  a  project  enters  development  with  poorly  defined  user  needs  and  requirements  the 
development  team  is  forced  to  clarify  requirements  and  conduct  expensive  re-work  to 
“get  the  project  right”  (Cooper  R.  G.,  1990,  p.  49  and  Repenning,  Gonsalves,  &  Black, 
2001).  Third,  it  is  important  to  conduct  a  rigorous  analysis  of  the  user  environment  be¬ 
cause  user  “insights  into  new  product  (and  process  and  service)  needs  and  potential  solu¬ 
tions  are  constrained  by  their  own  real-world  experience”  and  may  not  always  be  suffi¬ 
cient  for  new  product  development  (Von  Hippel,  1986,  p.  791).  Christensen  (2003)  sup¬ 
ports  this  observation  when  stating  that  companies  often  forgo  developing  disruptive 
technologies  because  their  customers  have  not  yet  recognized  the  opportunity  associated 
with  the  new  idea.  If  an  organization  wants  to  produce  a  novel  product  it  must  determine 
what  is  truly  valuable  to  its  customers,  which  cannot  always  be  detennined  by  asking  us¬ 
ers  (Kim  &  Mauborogne,  2005  and  Christensen,  2005). 

Ulrich  and  Eppinger  (2004)  stress  that  the  user  needs  are  independent  of  any 
product,  they  are  not  specific  to  any  concept,  and  that  the  best  way  to  capture  them  is 
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through  a  structured  process.  If  an  organization  makes  the  effort  to  understand  what  the 
user  needs  and  the  environment  in  which  the  user  operates,  the  development  team  will  be 
able  to  propose  a  solution  that  will  likely  satisfy  the  customer  needs  regardless  of  the 
technology  used.  This  understanding  of  how  the  product  will  be  used  and  what  is  of  val¬ 
ue  to  the  user  will  also  help  the  developer  analyze  the  cost/benefit  tradeoff  associated 
with  inevitable  changes  that  will  occur  due  to  changing  requirements  (Khurana  & 
Rosenthal,  1997).  A  dynamic  environment  is  an  inevitable  part  of  new  product  develop¬ 
ment  (Bacon,  Beckman,  Mowery,  &  Wilson,  1994)  and  understanding  user  needs  will 
help  an  organization  manage  the  change  (Khurana  &  Rosenthal,  1997).  The  necessity  to 
understand  fully  the  user  environment  and  the  stakeholder  environment  is  as  important,  if 
not  more  so,  in  the  Department  of  Defense  as  factors  that  drive  the  design  of  weapon  sys¬ 
tems  are  often  not  identified  in  the  official  requirement  documentation  (Gillespie,  2009). 

Developing  the  Product  Concept. 

Once  an  organization  has  identified  an  opportunity  and  has  collected  sufficient  in¬ 
formation  on  the  customer  and  user  needs,  it  must  develop  a  solution  to  meet  those  needs. 
It  is  at  this  point  where  Wheelwright  and  Clark’s  (1992)  development  funnel  is  expanded 
so  that  many  ideas  (or  concepts)  can  be  identified  for  evaluation.  These  different  con¬ 
cepts  compete  against  one  another  for  selection  by  proceeding  through  “screens”  similar 
to  Cooper’s  (1990)  gates.  Wheelwright  and  Clark  (1992)  suggest  the  initial  screen  not  be 
a  go/no-go  decision  but  an  evaluation  of  the  completeness  of  each  concept  to  determine 
which  additional  information  is  required  for  the  eventual  go/no-go  decision,  where  one, 
or  a  few,  of  the  best  concepts  will  be  selected  for  development.  Ulrich  and  Eppinger 
(2004)  and  the  Department  of  Defense  (DoD,  2008)  similarly  emphasize  the  importance 
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of  identifying  concepts  from  multiple  sources  for  evaluation.  As  stated  earlier,  Ulrich 
and  Eppinger  (2004),  Cooper  (1993),  and  Wheelwright  and  Clark  (1992)  suggest  the  con¬ 
cepts  should  be  evaluated  against  target  product  specifications  or  criteria  derived  from  the 
user  needs.  Khuranan  and  Rosenthal  (1997)  argue  the  product  specifications  should  be 
left  until  after  a  concept  is  selected  but  any  product  concept  should  be  “aligned  with  cus¬ 
tomer  needs”  (pg  106). 

Cooper  (1993)  contends  that  the  reason  the  development  of  a  product  concept  is 
so  important  is  that  the  steps  leading  up  to  and  including  the  development  of  a  product 
concept  are  “where  the  game  is  won  or  lost”  (pg  121).  Through  their  research,  Wheel¬ 
wright  and  Clark  (1992)  found  “the  outstanding  organization  starts  development  projects 
with  concept  development  on  a  firm  foundation”  (pg  15).  Ulrich  and  Eppinger  (2004) 
found  that  “the  degree  to  which  a  product  satisfies  customers  and  can  be  successfully 
commercialized  depends  to  a  large  measure  on  the  quality  of  the  underlying  concept”  (pg. 
98).  Ulrich  and  Eppinger  (2004)  state  that  while  a  good  concept  can  be  poorly  imple¬ 
mented  in  later  development  phases,  a  poorly  created  concept  rarely  leads  to  success. 

The  successful  development  of  the  product  concept  should  result  in  a  solution  that  meets 
the  needs  of  a  user  and  leaves  the  developer  with  a  reasonable  idea  of  resources  required 
to  develop  that  solution.  What  is  “reasonable”  is  detennined  by  the  level  of  project  defi¬ 
nition  (AACE  International,  1997;  AACE  International,  2005)  and  will  be  addressed  in  a 
later  chapter. 

Resource  Allocation. 

A  significant  insight  about  the  dynamics  of  product  development  is  “starting  a 
project  with  too  few  resources  almost  always  ensures  it  will  eventually  require  too  many” 
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(Black  &  Repenning,  2001,  p.  34).  Wheelwright  and  Clark  (1992)  found  that  companies 
can  over  commit  themselves  by  attempting  to  start  too  many  development  projects  and 
exceed  their  development  capacity  by  50-100%.  When  there  are  not  enough  resources  to 
commit  to  the  early  development  work,  due  to  either  over  commitment  of  resources  or 
poor  estimation  of  needed  resources,  the  activities  needed  to  define  the  project  can  be 
skipped,  which  leads  to  expensive  unanticipated  changes  later  in  development 
(Repenning,  Gonsalves,  &  Black,  2001).  It  is  impossible  to  remove  all  risk  from  product 
development  projects  since  they  are  risky  endeavors  by  their  very  nature  (Hillson, 
Effective  Opportunity  Management  for  Projects:  Exploiting  Positive  Risk,  2004).  How¬ 
ever,  by  allocating  enough  resources  early  in  the  development  cycle  to  develop  a  product 
concept  correctly,  an  organization  should  be  able  to  identify  risks  early,  make  plans  to 
address  them  (Bacon,  Beckman,  Mowery,  &  Wilson,  1994),  and  therefore  have  a  better 
understanding  of  the  cost  of  development  (Cooper  R.  G.,  1990). 
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Figure  2  -  Comparison  of  Development  Processes 

DoD  Product  Development  Process 

CJCSI  3170  and  the  DoDI  5000  series  describe  the  requirements  generation  and 
material  development  processes  by  which  the  Department  of  Defense  develops  the  sys¬ 
tems  it  needs  to  meet  the  needs  of  the  warfighting  community.  These  separate  processes 
must  work  in  concert  with  a  third  process  that  provides  the  resources:  the  Planning,  Pro¬ 
gramming,  Budgeting  and  Execution  (PPBE)  process.  The  overall  development  process 
begins  with  the  identification  of  user  needs,  continues  through  concept  genera¬ 
tion/selection,  design,  development,  and  deployment,  and  ends  with  disposal.  The 
process  is  designed  to  allow  potential  programs  to  originate  from  multiple  sources  but 
any  program  must  be  connected  to  a  set  of  user  needs  that  have  gone  through  the  JCIDS 
process.  Each  phase  of  the  development  process  is  separated  by  a  decision  review  that  is 
intended  to  act  as  a  “gate”  described  by  Cooper  (1990)  and  the  development  process  as  a 
whole  closely  mirrors  the  processes  found  in  product  development  literature  (Figure  2). 
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The  JCIDS  process  performs  the  role  of  the  commercial  marketing  function  that 
identifies  and  documents  the  user  needs  and  environment.  The  intended  role  of  the 
JCIDS  process  is  to  “identif[y]  and  [assess]  capability  needs  and  associated  performance 
criteria”  that  will  be  used  to  define  the  design  of  proposed  solutions  (CJCS,  2009,  pp.  A- 
2).  A  Capability  Based  Assessment  (CBA)  is  executed  to  identify  the  mission,  the  opera¬ 
tional  characteristics  and  attributes  of  the  desired  capability,  and  an  analysis  of  the  risk 
associated  with  the  desired  capability.  If  a  materiel  solution  is  required,  the  result  of  the 
CBA  will  also  include  a  recommendation  for  a  type  of  solution.  The  authority  to  validate 
the  capability  shortfall  found  in  the  CBA  process  and  to  certify  that  a  materiel  solution  is 
required  to  satisfy  the  need,  falls  to  the  Joint  Requirements  Oversight  Council  (JROC). 
Should  the  Council  agree  with  the  results  of  the  CBA,  an  Initial  Capabilities  Document 
(ICD)  is  created  from  the  information  gathered  during  the  CBA  and  it  becomes  the  initial 
requirements  and  specifications  that  any  proposed  material  solution  must  satisfy.  The 
final  go/no-go  decision  to  pursue  a  material  solution  falls  to  the  Milestone  Decision  Au¬ 
thority  (MDA)  at  the  Materiel  Development  Decision  (MDD)  review.  The  MDA  must 
detennine  if  enough  information  is  available  to  begin  the  assessment  of  potential  material 
solutions  (CJCS,  2009).  The  order  of  events  for  the  initial  part  of  the  DoD  process  is  dif¬ 
ferent  than  what  is  found  in  Wheelwright  and  Clark  (1992)  and  Ulrich  and  Eppinger 
(2004)  but  the  status  at  the  end  of  this  phase  is  the  same  in  both  cases.  For  each,  a  market 
opportunity  is  identified,  the  needs  of  users  are  identified,  and  the  organization  then  de¬ 
cides  if  it  is  beneficial  to  pursue  further  development. 

Once  the  decision  is  made  to  pursue  a  material  solution  Khurana  and  Rosenthal 
(1997),  Wheelwright  and  Clark  (1993),  and  Ulrich  and  Eppinger  (2004)  suggest  a  greater 


26 


understanding  of  user  needs  and  environment  is  needed.  Within  the  DoD  process  this 
effort  should  already  have  been  accomplished  by  the  JCIDS  process  (CJCS,  2009). 

When  the  MDA  decides  a  project  warrants  a  material  solution  the  project  enters  the  Ma¬ 
terial  Solution  Analysis  (MSA)  phase  through  the  Material  Development  Decision 
(MDD)  review  gate  (DoD,  DoD  Instruction  5000.02,  2008).  The  purpose  of  this  phase  is 
similar  to  the  purpose  of  the  second  phase  of  Wheelwright  and  Clark’s  (1992)  develop¬ 
ment  funnel:  to  develop,  assess,  and  select  a  potential  material  solution  for  a  go/no-go 
decision.  All  DoD  projects  must  pass  through  the  MDD  gate  but  DoD  Instruction  5000.2 
allows  a  program  to  proceed  to  any  phase  of  development  from  there.  A  program  that 
must  continue  through  the  MSA  phase  will  conduct  an  Analysis  of  Alternatives  to  “assess 
the  potential  material  solutions  to  satisfy  the  capability  need  documented  in  the  approved 
ICD”  (DoD,  DoD  Instruction  5000.02,  2008,  p.  15)  .  The  evaluations  conducted  in  the 
AoA  will  look  at  how  the  potential  material  solutions  meet  the  measures  of  effectiveness 
(similar  to  Burchill’s  (1993)  initial  product  specifications),  the  cost  and  schedule  to  de¬ 
velop  the  potential  material  solution,  how  the  solution  would  be  used,  and  the  overall  risk 
associated  with  the  potential  solution  (DoD,  DoD  Instruction  5000.02,  2008).  DoD  In¬ 
struction  5000.2  also  mandates  assessment  of  the  critical  technologies,  any  plans  to  ma¬ 
ture  and  test  that  technology,  and  the  manufacturing  feasibility  of  critical  technologies. 
The  type  of  infonnation  reviewed  during  the  AoA,  and  expected  to  be  found  in  the  poten¬ 
tial  material  solution  documentation,  is  of  the  same  type  found  in  commercial  product 
development  concepts:  needs,  technical  solution,  and  resources  required  to  develop  the 
solution. 


27 


Proposed  Definition  of  a  DoD  Product  Concept 

Though  neither  DoDI  5000.02,  CJCSI  3170.01G,  nor  the  associated  manual 
(CJCS,  Manual  for  the  Operation  of  the  Joint  Capabilities  Integration  and  Developement 
System,  2009),  contain  an  explicit  definition  of  the  DoD  equivalent  to  a  product  concept, 
each  requires  the  collection  of  some  or  all  of  the  information  found  in  a  commercial 
product  concept.  The  JCIDS  process  is  initiated  by  the  execution  of  a  CBA.  The  objec¬ 
tive  of  the  CBA  is  to  understand  and  document  user  needs,  conduct  an  analysis  of  which 
additional  capabilities  are  needed,  and  detennine  if  the  capabilities  can  be  met  by  a  non¬ 
material  solution  (CJCS,  CJCSI  3170.01G  -  Joint  Capabilities  Integration  and 
Development  System,  2009).  The  ICD  adds  to  the  infonnation  developed  in  the  CBA  by 
describing  the  operational  environment  in  which  the  capability  will  be  used  and  providing 
tasks  to  be  accomplished  with  associated  measurable  attributes  that  any  proposed  solution 
must  meet.  The  ICD  should  contain  all  the  user  needs  against  which  each  potential  solu¬ 
tion  will  be  evaluated  (CJCS,  2009).  DoDI  5000.02  addresses  the  other  two  types  of  in¬ 
formation  found  in  product  concepts:  solutions  and  needed  resources.  The  Analysis  of 
Alternatives  evaluates  each  proposed  material  solution  based  on  how  well  it  satisfies  the 
needs  documented  in  the  ICD  but  also  on  how  much  the  development  will  cost,  how  long 
the  development  will  be,  and  the  amount  of  risk  that  is  associated  with  each  material  so¬ 
lution.  The  Material  Solution  Analysis  phase  is  the  first  time  that  all  three  parts  of  a  con¬ 
cept  are  present.  The  purpose  of  this  phase  is  to  give  the  decision  makers  the  right  type 
and  level  of  fidelity  of  infonnation  to  make  the  decision  to  invest  money  in  the  develop¬ 
ment  of  a  solution  (DoD,  DoD  Instruction  5000.02,  2008),  which  is  the  same  purpose  as 
product  concept  generation  and  selection  (Wheelwright  &  Clark,  1992). 
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For  the  purpose  of  this  research  the  following  definition  will  be  used  to  describe  a 
DoD  materiel  concept:  a  solution  that  meets  the  needs  of  the  customers  and  users,  and 
identification  of  the  resources  required  to  develop  the  solution.  The  tenn  materiel  con¬ 
cept  is  used  in  place  of  product  concept  because  the  Department  of  Defense  does  not  de¬ 
velop  products  but  it  does  develop  materiel  solutions.  This  definition  subsumes  vague 
definitions  of  a  materiel  concept,  “potential  materiel  solution”  (AF  Center  For  Systems 
Engineering,  2009)  and  some  detailed  ones,  such  as  “a  collection  of  systems  and  their 
associated  organizational  elements  operating  in  accordance  with  a  CONOPS  to  provide  a 
needed  capability”  (Jacques,  2009).  However,  the  proposed  definition  implies  that  all 
three  types  of  infonnation  found  in  a  product  concept  should  be  contained  in  a  DoD  ma¬ 
teriel  concept. 

This  term,  materiel  concept,  and  the  working  definition  are  also  intentionally  va¬ 
gue  enough  to  be  applied  to  every  DoD  development  program  at  a  certain  level  of  ab¬ 
straction.  The  DoD  recognizes  that  because  no  two  development  programs  are  exactly 
alike  each  development  program  should  be  tailored  to  “fit  the  particular  conditions  of  that 
program”  (DoD,  2007).  Within  the  DoD  some  capabilities  may  be  satisfied  by  a  system 
or  system  of  systems  and  some  may  only  require  a  single  technological  solution.  This 
definition  can  be  applied  to  any  of  these  situations. 

Measuring  Concepts 

Ulrich  and  Eppinger  (2001)  state  that  the  process  of  selecting  a  concept  is  “the 
process  of  evaluating  concepts  with  respect  to  customer  needs  and  other  criteria,  compar¬ 
ing  the  relative  strengths  and  weaknesses  of  the  concepts,  and  selecting  one  or  more”  (pg 
124).  Repenning,  Gonsalves,  &  Black  (2001)  suggest  a  project  with  an  inadequate 


29 


concept  should  be  canceled  before  the  design  phase  and  Cooper  (1990)  states  that  the 
benefits  of  well-executed  front-end  activities  are  shorter  development  times  and  im¬ 
proved  success  rates.  However,  there  is  little  detailed  information  on  how  to  detennine  if 
a  concept  is  adequate  or  complete.  Burchill  (1993)  developed  a  multi-stage  process  to 
“engineer”  a  product  concept  to  help  guide  an  organization  through  the  difficult  task  but 
does  not  state  how  to  recognize  when  the  concept  is  complete.  His  process  utilizes  ana¬ 
lytical  tools  to  transfonn  the  infonnation  obtained  from  the  customer  into  a  product  con¬ 
cept  but  also  relies  a  great  deal  upon  experience. 

Perhaps  the  reason  for  such  little  information  on  how  to  judge  completeness  is 
that  it  truly  may  be  a  tacit  skill  acquired  over  time.  Other  reasons  on  why  there  is  little 
information  on  evaluating  a  concept’s  completeness,  or  maturity,  could  be  that  it  is  ex¬ 
tremely  difficult  to  develop  a  measure  for  such  a  wide  range  of  concepts  or  perhaps  cor¬ 
porations  have  a  method  to  measure  concepts  but  consider  the  information  proprietary. 
The  following  chapter  will  propose  a  framework  by  which  a  DoD  materiel  concept  may 
be  measured  to  detennine  if  it  is  mature  enough  to  proceed  through  a  decision  review. 
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III.  Process  for  Developing  a  Framework  for  Material 

Concept  Evaluation 

A  Four-Step  process  to  develop  a  framework 

This  chapter  describes  the  four  step  process  that  is  used  to  develop  a  framework 
by  which  decision  makers  could  evaluate  the  “completeness”  of  a  material  concept  and 
by  which  concept  developers  could  guide  their  development  efforts.  The  four  steps  are: 

1 .  Review  the  product  development  literature 

2.  Define  a  material  concept 

3.  Detennine  which  gates  in  the  DoD  process  should  be  used  as  evaluation  points 

4.  Identify  what  infonnation  is  required  by  a  decision  maker  at  each  gate 

The  steps  are  conducted  sequentially,  each  building  upon  the  work  conducted  in 
the  previous  step.  However,  the  entire  process  is  iterative  as  the  information  from  each 
step  is  checked  against  what  had  been  accomplished  in  previous  steps  to  ensure  that  it  is 
consistent  and  that  the  information  from  previous  steps  does  not  need  to  be  updated  due 
to  new  revelations.  The  information  for  the  framework  is  selected  base  upon  the  expe¬ 
riences  and  education  of  the  researchers  and  the  infonnation  found  in  the  literature.  Ad¬ 
ditionally,  the  framework  will  be  presented  to  domain  experts  for  their  review  and  to  eli¬ 
cit  feedback. 

Review  of  the  Product  Development  Literature 

In  addition  to  gaining  an  understanding  of  what  research  has  been  done  on  con¬ 
cept  maturity,  the  purpose  of  this  review  is  to  define  a  product  development  concept  and 
detennine  the  relative  influence  that  a  concept  has  on  the  development  process  outcome. 
Within  the  literature,  there  is  a  general  agreement  of  the  purpose  and  function  of  a  con- 
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cept  and  a  definition  was  created  for  a  DoD  material  concept,  which  is  detailed  in  chapter 
2.  Additionally,  a  preponderance  of  the  literature  indicates  a  mature  concept  is  necessary 
for  a  development  project  to  be  successful  but  it  will  not  guarantee  a  successful  outcome. 

A  vast  majority  of  the  literature  is  developed  for,  and  intended  to  be  applied  to, 
commercial  development.  An  analysis  to  compare  the  DoD  product  development  process 
to  the  generic  process  from  the  literature  is  detailed  in  chapter  2  and  the  result  is  that  the 
purpose  and  the  function  of  the  processes  are  nearly  identical. 

A  Definition  of  a  Material  Concept 

A  definition  of  a  DoD  product  concept  needs  to  be  created  for  the  research.  There 
are  two  reasons  for  developing  a  definition.  The  first  reason  is  to  develop  a  common  lan¬ 
guage.  Within  the  Department  of  Defense  the  term  “concept”  is  used  by  many  communi¬ 
ties  (including  the  acquisition  community)  and  each  has  a  unique  understanding  of  what 
the  term  means.  Additionally,  the  DOD  has  operating  concepts  and  concepts  of  opera¬ 
tions,  which  are  not  the  same  as  product  concepts.  By  articulating  what  a  DoD  product 
concept  is  and  what  information  it  contains,  semantic  arguments  should  be  kept  to  a  min¬ 
imum.  The  second  reason  for  developing  a  definition  is  that  it  is  difficult  to  evaluate,  or 
develop  a  framework  to  evaluate,  something  that  is  undefined. 

The  tenn  created  for  the  DoD  product  concept  is  “material  concept”  and  the  defi¬ 
nition  adopted  for  the  research  is:  a  solution  that  meets  the  needs  of  the  customers  and 
users,  and  identification  of  the  resources  required  to  develop  the  solution.  This  definition 
is  intended  for  DoD  use  but  it  is  created  to  incorporate  the  three  categories  of  infonnation 
found  in  a  concept  identified  from  the  literature  review:  needs,  solutions,  and  resources. 
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It  is  within  these  three  categories  that  any  infonnation  developed  for  the  concept  should 
be  classified. 

The  Concept  within  the  Stage-Gate  Process 

The  third  step  in  the  process  is  to  detennine  which  decision  gates  will  be  used  as 
events  where  the  material  concept  will  be  evaluated.  To  be  considered  for  selection,  a 
decision  gate  has  to  conform  to  three  criteria: 

1 .  The  gate  is  in  the  early  phases  of  the  development  process  where  a  concept  has 
the  greatest  influence  on  the  project 

2.  The  successful  transition  of  a  concept  through  the  gate  has  to  result  in  a  signifi¬ 
cant  increase  in  resources  to  further  develop  the  concept 

3.  There  is  a  gate  with  an  equivalent  purpose  found  in  the  generic  development 
process 

These  criteria  are  applied  to  the  decision  gates  found  in  the  DOD  development 
process  based  upon  how  guidance  documents  describe  the  purpose  and  function  of  each 
gate.  It  is  possible  that  the  purpose  and  function  of  the  decision  gates,  when  policy  is  put 
into  action,  will  differ  from  what  is  in  the  guidance.  However,  to  allow  for  traceability  of 
the  research  and  for  a  common  understanding,  only  the  guidance  documents  will  be  used 
to  determine  the  purpose  of  the  DoD  gates.  After  applying  these  criteria  to  the  DoD 
product  development  process  three  gates  emerged  as  acceptable  points  to  evaluate  the 
material  concept.  The  three  gates  were  1)  the  Air  Force/Joint  Requirements  Oversight 
Council  (AFROC/JROC)  where  the  user  needs  and  capability  deficiencies  are  validated 
and  documented  in  an  Initial  Capabilities  Document  (ICD);  2)  the  Material  Development 
Decision  where  initial  solutions  to  meet  the  identified  need  are  screened;  and  3)  Miles¬ 
tone  A  where  a  concept  is  selected  for  further  development. 
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Determine  the  Information  Required  for  a  Gate 

The  final  phase  is  to  determine  what  infonnation  a  decision  maker  requires  to  de¬ 
termine  the  effectiveness  and  suitability  of  the  concept  to  meet  user  needs.  Using  the 
principles  of  systems  modeling  (Maier  &  Rechtin,  2002)  and  system  requirement  defini¬ 
tion  (Blanchard  &  Fabrycky,  2006),  a  series  of  questions  are  asked  at  each  decision  gate 
to  scope  the  type  of  information  that  is  being  considered.  The  questions  are: 

1 .  What  is  the  purpose  of  this  gate  from  the  perspective  of  the  decision  makers? 

2.  What  activities  occur  following  this  gate? 

3.  What  infonnation  will  allow  the  decision  maker  to  determine  the  effectiveness 
and  suitability  of  the  concept  and  if  it  is  ready  for  the  next  phase  of  develop¬ 
ment? 

4.  What  information  is  required  to  allow  those  developing  the  project  in  the  fol¬ 
lowing  phase  to  be  successful  in  their  activities? 

5.  How  might  the  context  affect  the  type  or  amount  of  information  needed? 

6.  Which  Department  of  Defense  Architecture  Framework  (DoDAF)  views  are 
useful  to  present  the  information? 

As  a  final  check,  information  identified  from  the  series  of  questions  had  to  belong 
to  one  of  the  three  categories  of  information  found  in  a  concept:  needs,  solutions,  or  re¬ 
sources.  Additionally,  only  infonnation  that  is  required  to  develop  and  mature  the  con¬ 
cept  will  be  selected  for  the  framework.  Infonnation  already  required  at  a  decision  gate 
by  the  guiding  documents,  which  has  no  value  to  a  developing  concept,  is  excluded  from 
the  framework. 

The  views  of  the  Department  of  Defense  Architecture  Framework  (DoDAF)  are 
used  to  present  the  information  because  the  results  of  this  research  are  intended  to  be 
used  within  the  DoD  acquisition  process.  However,  a  different  architecture  framework, 
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like  the  Zachman  enterprise  architecture  framework  (Zachman,  1987),  can  be  used  to 
present  the  information. 

Using  these  four  steps,  a  framework  is  created  that  gives  decision  makers  and 
concept  developers  a  “yardstick”  and  a  common  language  to  evaluate  a  concept  and  de¬ 
termine  if  the  concept  is  mature  enough  to  proceed  to  the  next  stage  of  development.  The 
framework  is  discussed  in  detail  in  the  following  chapter. 
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IV.  Framework  for  Evaluation 


Context  of  the  Framework 

The  framework  developed  in  this  chapter  is  intended  to  be  used  for  the  evaluation 
of  material  concepts  within  the  fuzzy  front-end  of  the  product  development  process, 
which  includes  all  the  actions  up  to  and  including  concept  selection.  Within  the  Depart¬ 
ment  of  Defense  (DoD)  acquisition  process  this  would  be  the  actions  that  occur  up  to  and 
including  the  decision  made  at  Milestone  A.  By  this  point  in  both  DoD  and  non-DoD 
product  development,  when  a  concept  is  selected  for  development,  the  decisions  that  de¬ 
termine  50-80%  of  the  life  cycle  cost  of  the  product  are  already  made  (Wheelwright  & 
Clark,  1992,  Kaminski  &  Lyles,  2008  and  LtGen  Shakleford,  2009).  Additionally,  once 
a  concept  is  selected  for  development  an  organization  will  then  apply  considerable  re¬ 
sources  to  turn  the  concept  into  a  product.  The  work  conducted  in  this  front-end  of  de¬ 
velopment  has  a  great  impact  on  the  success  or  failure  of  the  following  development. 
However,  as  was  noted  in  a  previous  chapter  there  is  little  guidance  to  detennine  if  the 
work  performed  in  this  phase  is  adequate  or  the  concept  developed  in  this  phase  is  com¬ 
plete.  The  framework  developed  in  this  chapter  is  intended  to  be  used  as  a  common  lan¬ 
guage  and  yardstick  to  evaluate  the  maturity  of  material  concepts  in  a  stage-gated  product 
development  process.  When  measured  against  this  yardstick,  a  decision  maker  should  be 
able  to  detennine  if  the  concept  contains  the  right  type  of  information  and  enough  of  it  to 
meet  a  decision  gate.  However,  this  framework  is  not  intended  to  evaluate  if  the  concept 
is  a  good  solution,  only  that  it  has  been  well  thought  out. 

The  framework  detailed  in  this  chapter  was  developed  with  the  process  described 
in  chapter  3.  The  decision  gates  were  identified  first.  Then  the  information  a  decision 
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maker  requires  at  each  gate  was  identified.  Finally,  Department  of  Defense  Architecture 
Framework  (DoDAF)  views  were  selected  that  presented  the  information.  However,  for 
the  sake  of  readability,  the  recommended  DoDAF  2.0  views  are  in  parentheses  and  ac¬ 
company  the  description  of  the  infonnation.  Table  1  gives  a  description  of  the  DoDAF 
2.0  views. 
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Table  1  -  DoDAF  Views  (  DoD  Deputy  Chief  Information  Officer) 


Description 

View 

Description 

View 

All  View 

AV 

A  mapping  of  system  functions  (ac- 

SV- 

Operational  View 

ov 

tivities)  back  to  operational  activi- 

5a 

Systems  View 

sv 

ties  (activities) 

Services  View 

SvcV 

Capability  View 

cv 

Standards  View 

StdV 

Describes  a  Project's  Visions,  Goals,  Ob- 

AV-1 

A  mapping  of  systems  back  to  ca- 

SV- 

jectives,  Plans,  Activities,  Events,  Condi- 

pabilities  or  operational  activities. 

5b 

tions.  Measures,  Effects  (Outcomes),  and 

produced  objects. 

The  high-level  graphical/textual  descrip- 

OV-1 

The  emerging  technologies,  soft- 

SV-9 

tion  of  the  operational  concept. 

ware/hardware  products,  and  skills 

that  are  expected  to  be  available  in  a 

given  set  of  timeframes  and  that  will 

affect  future  system  development. 

A  description  of  the  resource  flows  ex- 

OV-2 

The  identification  of  services,  ser- 

SvcV 

changed  between  operational  activities. 

vice  items,  and  their  interconnec- 

-1 

tions 

The  organizational  context,  role  or  other 

OV-4 

A  description  of  resource  flows  be- 

SvcV 

relationships  among  organizations 

tween  services 

-2 

The  capabilities  and  activities  (operational 

OV- 

The  relationships  among  and  be- 

SvcV 

activities)  organized  in  a  hierarchal  struc- 

5a 

tween  systems  and  services  in  a 

-3a 

ture. 

given  architecture 

The  context  of  capabilities  and  activities 

OV- 

The  functions  performed  by  services 

SvcV 

(operational  activities)  and  their  relation- 

5b 

and  the  service  data  flows  among 

-4 

ships  among  activities,  inputs,  and  outputs 

service  functions 

One  of  three  models  used  to  describe  op- 

OV- 

A  mapping  of  services  back  to  oper- 

SvcV 

erational  activity.  It  identifies  business 

6a 

ational  activities 

-5 

rules  that  constrain  operations 

The  identification  of  systems,  system 

SV-1 

A  hierarchy  of  capabilities  which 

CV-2 

items,  and  their  interconnections 

specifies  all  the  capabilities  that  are 

referenced  throughout  the  architec- 

tural  descriptions 

A  description  of  resource  flows  between 

SV-2 

The  dependencies  between  planned 

CV-4 

systems 

capabilities  and  the  definition  of 

logical  groupings  of  capabilities 

The  relationships  among  systems  in  a  giv- 

SV-3 

A  mapping  between  the  capabilities 

CV-6 

en  Architectural 

required  and  the  operational  activi- 

Description.  It  can  be  designed  to  show 

ties  that  those  capabilities  support 

relationships  of  interest, 

(e.g.,  system-type  interfaces,  planned  vs. 

existing  interfaces). 

The  functions  (activities)  performed  by 

SV-4 

The  listing  of  standards  that  apply  to 

StdV 

systems  and  the  system  data  flows  among 

solution  elements 

-1 

system  functions  (activities). 
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Assessing  Maturity 

If  the  purpose  of  concept  development  is  to  give  a  decision  maker  the  right  type 
and  level  of  information  to  make  an  investment  decision,  then  what  is  the  right  type  and 
level?  Rephrased  the  question  becomes:  when  is  a  concept  mature  enough  to  be  consi¬ 
dered  for  selection?  Wheelwright  and  Clark  (1992)  contend  only  concepts  that  meet  user 
needs  and  are  complete  should  be  selected.  However,  they  do  not  elaborate  on  how  to 
identify  when  a  concept  is  complete.  This  research  indicates  that  elements  common  to 
any  materiel  concept  can  be  evaluated  with  a  developmental  scale.  These  elements  are 
pieces  of  infonnation  required  to  develop  the  three  types  of  information  found  in  a  ma¬ 
terial  concept  as  defined  in  Chapter  2:  needs,  solution,  and  resources.  The  evaluation  of 
these  elements  should  indicate  if  a  concept  is  mature  at  a  stage  of  development  relative  to 
where  it  should  be.  The  proposed  model  herein  is  similar  to  a  model  used  to  evaluate  ear¬ 
ly  childhood  development. 

The  Child  Development  Inventory  (CDI)  is  a  300-item  questionnaire  that  assesses 
the  development,  symptoms,  and  behavior  problems  of  young  children  (Ireton,  1995). 

The  developmental  scales  measure  social,  self-help,  gross  motor,  fine  motor,  expressive 
language,  language  comprehension,  letters,  numbers,  and  general  development.  The 
measures  within  each  of  these  categories  for  a  mature  2-year-old  will  be  different  from 
those  for  a  mature  4-year-old,  but  both  children  could  be  considered  mature  relative  to 
their  stage  of  development. 

Similarly,  a  materiel  concept  that  is  in  the  initial  stage  of  development  will  have 
less  detailed  information  than  a  concept  being  considered  for  selection  but  both  could  be 
considered  mature  for  their  relative  stages  of  development.  The  reality  of  any  develop- 
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ment  process  is  the  decision  maker  will  be  the  ultimate  authority  in  determining  when  a 
concept  is  mature  enough  for  selection.  However,  a  framework  that  measures  the  maturi¬ 
ty  of  elements  of  a  materiel  concept,  relative  to  its  stage  of  development,  could  be  used  as 
a  benchmark  for  decision  makers  and  as  a  guide  for  those  who  generate  concepts.  There 
are  categories  of  maturity  elements,  which  are  found  within  systems  engineering,  which 
will  need  to  be  addressed  in  a  materiel  concept.  Though  these  elements  are  defined  as 
part  of  systems  engineering,  the  research  also  indicates  systems  architecture  elements  and 
risk  management  elements  require  specific  attention  because  of  the  infonnation  they  con¬ 
tain  that  decision  makers  need  to  execute  the  “go/kill/hold/recycle”  decision  during  the 
front-end  of  the  stage-gated  development  process  (Cooper  R.  G.,  1990,  p.  46). 

Concept  Evaluation  and  Selection  within  a  Stage-Gated  Process 

The  general  purpose  of  every  decision  gate  in  the  front  end  is  to  prevent  any  con¬ 
cept  from  continuing  to  a  development  phase  before  it  is  ready.  At  the  end  of  a  develop¬ 
ment  phase,  the  development  team  needs  to  demonstrate  to  the  decision  maker  that  the 
concept  is  developed  enough  to  proceed  to  the  next  phase  and  that  further  development  of 
the  concept  will  benefit  the  organization  and  the  intended  user.  The  elements  used  to  as¬ 
sess  and  mature  a  concept  should  define  the  level  of  robust  early  planning  required  at  a 
given  decision  point.  A  simple  way  to  understand  the  appropriate  time  for  any  particular 
element  is  to  relate  the  purpose  of  the  element  to  the  specific  objective  of  the  decision 
following  a  development  stage.  A  descriptive,  stage-gated  process  based  upon  processes 
described  in  the  product  development  literature  and  the  one  used  by  the  Department  of 
Defense  (DoD)  is  presented  here  (figure  6,  pg.  55).  While  the  forms  of  the  processes 
may  differ,  the  purpose  and  function  of  the  DoD  process  closely  mirrors  that  of  the 
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process  found  in  the  literature  and  will  be  used  as  a  baseline.  Both  begin  with  the  con¬ 
cept  development  phase. 

For  the  practitioner  within  the  DoD  and  commercial  development,  it  is  important 
to  understand  the  considerations  and  discriminators  for  each  progressive  investment  deci¬ 
sion  in  the  concept  development  process.  The  first  gate.  Opportunity  Identification,  oc¬ 
curs  when  the  Joint  Requirements  Oversight  Council  (JROC)  approves  and  validates  an 
Initial  Capabilities  Document  (ICD)  (CJCS,  CJCSI  3170.01G  -  Joint  Capabilities 
Integration  and  Development  System,  2009),  which  contains  originating  requirements. 
The  analysis  conducted  to  develop  an  ICD  should  identify  the  shortfall  in  military  capa¬ 
bility,  identify  the  user  needs,  and  detennine  that  a  new  development  product  is  required 
to  meet  the  need.  The  JROC  must  determine  if  this  development  opportunity  that  was 
identified  from  a  market  environmental  analysis  is  adequately  important  for  the  allocation 
of  resources. 

If  the  JROC  detennines  that  the  ICD  is  complete  and  that  it  identifies  a  valid 
need,  the  next  gate,  Concept  Screening,  occurs  at  the  Materiel  Development  Decision 
(MDD).  The  primary  purpose  of  the  MDD  is  to  act  as  the  official  entry  gate  to  the  mate¬ 
riel  development  process.  It  also  acts  as  a  filter  to  prevent  the  concepts  that  are  infeasible 
for  development,  or  do  not  meet  the  need  identified  in  the  ICD,  from  progressing  any  fur¬ 
ther  in  the  development  process.  Any  concept  that  the  decision  makers  deem  sufficient 
will  undergo  further  maturation  with  deeper  analysis  in  the  Materiel  Solution  Analysis 
phase. 

The  third  screening  gate,  Concept  Selection,  occurs  at  Milestone  A.  At  this  in¬ 
vestment  decision,  a  concept  is  selected  based  upon  a  set  of  criteria.  The  criteria  should 
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include  user  needs,  risk  associated  with  development,  cost  to  develop,  operate  and  sustain 
the  solution,  and  the  benefits  the  development  brings  to  the  organization.  A  concept  that 
demonstrates  that  it  meets  the  criteria  and  is  selected  will,  assumedly,  have  resources  al¬ 
located  to  conduct  preliminary  design.  Later  gates  with  detailed  design  and  fabrica¬ 
tion/production  readiness  will  be  necessary,  but  substantial  policy  and  guidance  for  these 
later  gates  exists  for  the  DoD  (DoD,  DoD  Instruction  5000.02,  2008)  and  will  not  be  ad¬ 
dressed  further  in  this  research.  However,  it  must  be  understood  that  the  information  col¬ 
lected  for  these  early  decision  gates  will  greatly  affect  and  support  the  activities  of  the 
following  development  phases. 

In  this  research,  early  decision  gates  define  major  milestones  that  are  used  to  pre¬ 
vent  any  concept  from  progressing  to  a  phase  of  development  before  it  is  ready.  A  con¬ 
cept  can  pass  through  a  decision  gate  if  the  products  for  the  current  phase  of  work  are 
complete  and  if  the  decision  authority  determines  that  there  is  benefit  to  further  develop¬ 
ment.  The  worthiness  of  a  concept  for  additional  development  is  dependent  upon  contex¬ 
tual  issues  like  resource  constraints  and  political  climate,  in  addition  to  a  concept’s  ma¬ 
turity.  A  development  team  has  no  control  over  the  contextual  issues  but  it  can  ensure 
the  proper  definition  and  analysis  associated  with  a  decision  gate  has  been  completed. 

The  following  sections  attempt  to  define  the  key  criteria  required  to  evaluate  a  concept’s 
maturity  at  the  three  decision  gates.  In  an  effort  to  mature  a  concept  to  the  point  of  selec¬ 
tion,  the  practitioner  should  develop  and  accrue  a  robust  set  of  maturity  elements  that, 
when  combined,  will  provide  sufficient  infonnation  to  the  investment  decision  maker  and 
will  provide  the  foundation  upon  which  the  remainder  of  the  project  will  be  built. 
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Gate  1:  Opportunity  Identification  -  ICD  Approval  (AFROC/JROC) 

The  first  steps  in  product  development  are  the  identification  of  an  opportunity  and 
then  an  initial  identification  of  user  needs  (Ulrich  &  Eppinger,  Product  Design  and 
Development,  2004).  If  the  organization  decides  the  opportunity  is  worth  pursuing,  it 
conducts  a  rigorous  analysis  of  the  user  needs.  The  order  of  activities  in  the  DoD  process 
is  slightly  different  because  the  rigorous  analysis  of  user  needs  is  conducted  before  the 
decision  to  pursue  an  opportunity  (CJCS,  CJCSI  3 170.0 1G  -  Joint  Capabilities 
Integration  and  Development  System,  2009).  However,  at  the  end  of  this  phase  an  organ¬ 
ization  in  either  environment  will  have  identified  an  opportunity,  identified  user  needs 
and  then  must  decide  if  it  is  beneficial  to  pursue  further  development. 

The  Joint  Capability  Integration  and  Development  System  (JCIDS)  instruction  re¬ 
quires  analysis  be  conducted  to  identify  and  validate  gaps  in  military  capability,  to  cha¬ 
racterize  the  risk  associated  with  the  gap,  and  detennine  what  type  of  solution  is  required 
to  fill  the  gap.  Those  conducting  the  analysis  must  identify  how  things  are  currently 
done,  deficiencies  in  capability,  what  causes  the  deficiencies,  objectives  to  be  met,  tasks 
to  be  done,  and  operational  performance  criteria  associated  with  the  tasks.  The  purpose 
of  this  phase  is  to  determine  key  questions  regarding  the  customer  needs,  such  as  “who 
needs  it,  why,  how  might  they  use  it,  what  they  need  to  do  and  when  do  they  need  it.”  If 
the  analysis  detennines  that  a  materiel  solution  is  required  to  satisfy  the  capability  defi¬ 
ciency,  the  analysis  team  documents  the  results  in  the  ICD  and  then  presents  it  to  the 
JROC  for  approval.  The  JROC  must  detennine  if  the  military  deficiency  warrants  a  solu¬ 
tion  and  that  the  solution  must  be  a  new  development  product. 
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Information  Elements  for  Gate  1. 


The  ICD  currently  has  no  requirements  for  architecture  products  beyond  a  Con¬ 
cept  of  Operations  (CONOPS)  and  an  associated  Concept  Graphic  (OV-1).  However, 
there  are  several  architecture  maturity  elements  that  should  be  used  to  support  the 
JROC’s  decision  at  this  gate  (figure  ),  and  they  can  be  developed  from  the  documentation 
and  infonnation  currently  required  in  the  generation  of  the  Initial  Capabilities  Document. 
The  Joint  Ops  Concepts  and  CONOPS  (defined  in  AFPD  10-28  or  IEEE  Std  1362-1998) 
can  be  used  to  detennine  what  the  users  need  to  do  and  how  they  expect  to  do  it.  A  well 
defined  CONOPS  identifies  the  mission  area,  timeframe,  assumptions  with  regards  to 
projected  capabilities  of  ourselves  and  our  adversaries,  desired  effects  and  both  the  ne¬ 
cessary  and  supporting  capabilities  that  are  needed.  This  information,  as  well  as  other 
information  contained  within  a  CONOPS,  can  be  used  to  develop  an  overview  of  the  sys¬ 
tem  architecture  products  that  will  characterize  the  infonnation  associated  with  the  de¬ 
sired  capability  (AV-1).  The  CONOPS  should  also  include  sequenced  actions  for  the  op¬ 
erational  and  support  scenarios  envisioned  by  the  user.  Using  tools  such  as  Use  Case 
modeling  or  traditional  functional  decomposition,  this  infonnation  can  be  captured  in  op¬ 
erational  activity  models  (e.g.,  OV-5a/b  in  the  DoDAF  (DoDAF  2.0,  2009))  which  show 
what  must  be  done  and  gives  the  context  of  how  it  might  be  done.  Any  known  rules  or 
constraints  that  may  restrict  operations  should  be  captured  (e.g.,  OV-6a  in  DoDAF)  to 
give  a  better  understanding  of  the  user  environment.  In  addition,  the  identification  of  any 
organizations  involved  in  the  activities  (OV-4)  and  resources  that  flow  between  activities 
(OV-2)  will  help  characterize  the  situation  for  the  design  teams  in  future  phases.  Finally, 
the  capabilities  associated  with  the  mission  (CV-2)  and  how  those  capabilities  support  or 
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interact  with  other  operational 
activities  (CV-6)  and  with  each 
other  (CV-4)  should  be  cap¬ 
tured. 

Any  existing  systems 
and/or  services  (e.g  U.S.  Air 
Force)  involved  with  the  desired 
capability  described  in  the  ICD 
should  be  identified  (SV-1, 


Opportunity 

Identification 


SvcVl)  and  their  interactions  should  be  characterized.  If  further  definition  of  the  interac¬ 
tion  and  various  systems  and  services  associated  with  the  concept  is  warranted,  resource 
flows  and  existing/planned  interfaces  can  be  identified  (SV-2,3,  SvcV-2,3).  In  order  to 
determine  gaps  between  needed  capability  (as  defined  by  the  operational  activity  models, 
e.g.,  OV-5)  and  current  system  capability,  system  and  services  functionality  descriptions 
can  be  developed  for  current  systems  (SV-4,  SvcV-4)  and  mapped  to  required  operational 
activities  using  traceability  matrices  (SV-5,  SvcV-5).  If  necessary  during  this  early  needs 
identification  phase,  these  systems  and  services  architecture  elements  would  be  restricted 
to  existing  systems/services  (for  the  time  frame  of  interest),  and  would  contain  only  the 
detail  necessary  to  identify  the  projected  operational  gaps  and  determine  the  reason  for 
the  gaps. 

It  should  be  noted  that  many  of  these  architecture  products  are  or  may  be  required 
at  later  gates  associated  with  the  DoD  acquisition  process,  but  early  collection  of  the  in¬ 
formation  and  definition  of  these  products  during  the  needs  identification  phase  will  help 
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in  the  long  tenn  effort.  The  initial  development  of  these  architecture  elements  before  the 
first  decision  gate  serves  three  purposes.  First,  the  methodical  development  of  the  ele¬ 
ments  can  be  used  to  document  existing  capability,  clarify  any  gaps  in  the  capability,  and 
characterize  the  operational  risk  associated  with  the  gap.  Second,  the  insight  gained  from 
these  elements  can  aid  in  assessing  the  form  of  solution  to  meet  the  capability  gap.  Last¬ 
ly,  the  elements  serve  as  the  foundation  for  future  development  phases.  Each  proposed 
solution  will  be  designed  and  evaluated  based  upon  requirements  developed  from  the 
ICD. 

A  very  important  component  of  the  ICD  that  is  not  currently  being  adequately  ad¬ 
dressed  is  that  of  effectiveness  measures.  As  part  of  the  needs  identification  process, 
needed  capabilities  should  be  identified  in  terms  of  tasks,  attributes  and  measures.  The 
measures  at  this  level  are  best  described  as  mission  level  Measures  of  Effectiveness 
(MOE’s).  While  the  JCIDS  policy  has  always  required  the  inclusion  of  MOE’s  in  the 
ICD,  recent  reports  have  suggested  that  ICD’s  are  not  adequately  addressing  how  the  op¬ 
erational  needs  are  to  be  quantified  for  subsequent  evaluation  of  alternatives  (Sadauskas, 
2008).  The  MOE’s  serve  a  similar  purpose  as  the  initial  target  specifications  found  in  the 
product  development  literature,  which  is  to  guide  the  development  and  selection  of  poten¬ 
tial  solutions.  Identification  of  MOE’s  is  critical  to  the  concept  maturation  process,  and 
is  included  herein  as  one  of  the  maturity  elements. 

One  final  maturity  element  that  is  already  required  by  CJCSI  3 170.0 1G  (2009) 
and  should  be  developed  in  this  early  phase  is  an  operational  risk  assessment.  This  risk 
assessment  describes  the  risk  of  not  filling  the  operational  need.  In  DoD  terms,  this  could 
be  higher  projected  loss  rates,  greater  numbers  of  personnel  and  systems  allocated  to  mis- 
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sions,  projected  lengthening  of  the  campaign  duration,  and/or  increased  vulnerability  due 
to  insufficient  deterrent  capabilities.  In  later  stages,  these  operational  risks  will  be 
weighed  against  the  cost  and  technical  risk  associated  with  pursuing  a  materiel  solution. 

How  Much  is  Enough? 

As  mentioned  in  a  previous  section  of  this  chapter  a  concept  should  be  measured 
relative  to  its  phase  of  development.  In  this  early  phase  it  is  difficult  to  state  how  much 
information  is  enough  as  the  capabilities  that  may  be  brought  to  the  JROC  can  range  from 
a  completely  new  need  that  has  never  been  done  to  a  capability  with  a  long  history  that 
requires  updating.  In  any  case  enough  infonnation  must  be  collected  and  enough  analysis 
done  to  show  what  is  of  value  to  the  user,  how  the  user  currently  operates,  and  why  this 
way  is  not  sufficient  to  do  what  they  need.  If  these  things  can  be  demonstrated  and  do¬ 
cumented  the  decision  maker  should  have  enough  information  to  detennine  that  the  con¬ 
cept  is  mature  enough  for  the  next  phase  of  development. 

Gate  2:  Concept  Screening  -  MDD  (MDA) 

After  an  organization  decides  to  pursue  a  development  opportunity,  they  should 
identify  as  many  potential  solutions  as  possible.  These  ideas  should  be  developed,  com¬ 
bined  and  discarded  as  they  pass  through  a  series  of  screens  so  that  only  a  few  of  the  best 
ideas  remain  for  consideration  (Wheelwright  &  Clark,  1992).  The  DoD  calls  this  screen 
Materiel  Development  Decision  and  uses  it  as  a  final  check  of  the  JROC  recommendation 
to  allow  the  further  development  of  a  materiel  solution  (DoD,  2008).  The  decision  maker 
at  MDD  is  called  the  Milestone  Decision  Authority  (MDA).  The  MDA  reviews  the  ap¬ 
proved  ICD  and  any  proposed  concepts  to  ensure  that  the  material  solution  decision  has  a 
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solid  foundation,  is  based  on  justified  information,  and  can  be  developed  within  time  and 
resource  constraints.  If  the  concepts  are  deemed  adequately  mature,  they  proceed  to  the 
next  phase  of  development  where  they  are  further  explored.  In  order  to  ensure  that  the 
approved  concepts  are  adequately  mature,  and  in  an  effort  to  encourage  more  rigor  at  this 
early  gate,  the  MDA  needs  important  pieces  of  information  defined  herein  as  key  concept 
maturity  elements. 

Information  Elements  for  Gate  2. 

The  information  needed  by  MDA  at  Concept  Screening  is  largely  associated  with 
development  of  new  or  modified  systems  included  in  the  proposed  concepts  (figure  4). 
These  concepts  involve  legacy  systems  and  any  anticipated  changes  in  operations  and/or 
materiel  to  existing  systems  should  be  identified.  A  critical  piece  of  information  for  the 
decision  maker  at  this  stage  is  the  scope  of  the  required  changes  to  implement  the  con¬ 
cept,  since  later  decision  gates  will  be  increasingly  associated  with  development  of  the 
individual  component  systems  of  the  concept.  If  the  full  scope  of  the  concept  is  not  fully 
understood  prior  to  a  system  development  decision,  either  the  full  operational  capability 
will  not  be  realized,  or  significant  cost  impacts  will  be  forthcoming  to  address  needed 
modifications  to  other  systems. 
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At  this  stage  of  concept  development  the  actual  proposed  solutions  are  explained 
in  terms  of  the  required  functionality  (SV-4,  SvcV-4),  and  the  relationship  between  the 
need  and  solution  is  defined  for  the  associated  systems  (SV-5,  SvcV-5).  These  architec¬ 
ture  elements  may  have  been  initially  defined  for  existing  systems  during  the  needs  iden¬ 
tification  process  associated  with  Gate  1 ,  but  they  will  need  to  be  updated  and  augmented 
for  the  envisioned  modifications  and/or  developmental  systems  associated  with  a  pro¬ 
posed  concept.  The  interfaces  and  relation¬ 
ships  between  systems  and  services  should  also 
be  updated  (SV-1-3,  SvcV-1-3).  The  technol¬ 
ogies  critical  to  the  solution  need  to  be  identi¬ 
fied  (SV-9)  and  any  known  standards  applica¬ 
ble  to  the  concept  (StdV-1)  should  be  captured. 

The  level  of  detail  required  for  any  ar¬ 
chitecture  element  at  this  point  should  be  dri¬ 
ven  by  the  decision  at  hand  and  the  decision 
maker.  At  the  concept  screening  gate,  scope 
and  problem  definition  dominate  the  decision 
objectives,  and  the  architecture  definition  to  support  this  decision  will  likely  require  no 
more  than  subsystem  identification  for  the  component  systems  of  the  concept.  Indeed, 
novel  solutions  considered  at  the  concept  screening  gate  will  not  likely  support  definition 
below  this  level.  Even  existing  solutions  where  detailed  information  is  available  will  not 
require  all  this  detail  be  included  in  the  architecture  products  at  this  early  stage.  The 


Figure  4  -  Information  Elements  for  Gate  2 
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goals  of  this  phase  are  to  conduct  the  analyses  to  show  the  proposed  solution  will  meet 
the  identified  needs  and  to  identify  and  characterize  the  risks  associated  with  the  solution. 

Concepts  that  are  allowed  to  proceed  through  the  screening  gate  will  need  to  un¬ 
dergo  further  definition  and  will  eventually  have  to  compete  against  each  other.  The 
competition  is  in  the  form  of  a  cost/benefit  analysis  and  the  criteria  against  which  the 
concepts  are  measured  should  be  identified  prior  to  the  concept-screening  gate.  Quantita¬ 
tive  target  specifications  (Measures  of  Performance)  that  describe  system  characteristics 
should  be  developed  prior  to  the  gate  for  use  in  the  analysis.  The  DoD  conducts  an  Anal¬ 
ysis  of  Alternatives  (AoA)  during  the  concept  refinement  phase  that  acts  as  the 
cost/benefit  analysis.  The  AoA  study  plan  sets  the  parameters  for  the  critical  technolo¬ 
gies  and  cost  drivers,  and  decision  objectives  for  the  AoA.  Sufficient  risk  identification 
associated  with  the  technologies,  interfaces,  or  changes  to  existing  systems  should  be 
completed  to  ensure  that  the  AoA  further  addresses  all  pertinent  issues. 

According  to  the  AoA  Handbook  (Office  of  Aerospace  Studies,  2008)  the  AoA 
study  plan  should  describe  how  the  following  questions  will  be  answered  in  the  subse¬ 
quent  Materiel  Solutions  Analysis  phase: 

•  What  alternatives  provide  validated  capabilities? 

•  Are  the  alternatives  operationally  effective  and  suitable? 

•  Can  the  alternatives  be  supported? 

•  What  are  the  risks  (technical,  operational,  programmatic)  for  each  alternative? 

•  What  are  the  life-cycle  costs  for  each  alternative? 

•  How  do  the  alternatives  compare  to  one  another? 

Due  to  the  uncertainty  associated  with  any  new  development,  risks  will  still  be 
present  in  a  program  or  project  regardless  of  the  level  of  risk  management.  However,  if 
an  ample  amount  of  rigor  is  applied,  the  more  expensive  risks  can  be  identified  and  their 
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impacts  minimized.  Further,  if  risks  are  identified  and  sound  risk  mitigation  plans  are 
developed,  a  much  more  realistic  idea  of  the  resources  required  to  complete  development 
will  be  produced,  thereby  resulting  in  a  more  realistic  cost  estimate.  It  is  during  the  risk 
identification  phase  when  the  important  relationships  between  systems  engineering  and 
system  architecture  are  defined.  Risk  management  is  the  process  used  to  manage  the  un¬ 
certainty  associated  with  new  development  (Hillson,  2004)  and  therefore,  risk  manage¬ 
ment  is  a  critical  element  of  concept  maturity. 

Gate  3:  Concept  Selection  -  Milestone  A  (MDA) 

The  purpose  of  the  concept  selection  gate  is  to  evaluate  proposed  concepts  with 
respect  to  customer  needs  and  to  the  resources  required  to  develop  the  concept  (Ulrich  & 
Eppinger,  Product  Design  and  Development,  2004).  Again,  this  is  similar  to  the  DoD 
process.  In  2008,  the  National  Research  Council  (NRC)  published  a  SAF/AQR  commis¬ 
sioned  study  entitled  “Pre -Milestone  A  and  Early  Phase  System  Engineering:  A  retros¬ 
pective  Review  and  Benefits  for  Future  Air  Force  Systems  Acquisition.”  This  study  ex¬ 
amined  the  important  elements  of  early  systems  engineering  activities  and  their  applica¬ 
tion  to  address  the  root  causes  of  program  failure  during  the  developmental  phases  before 
the  major  financial  investment  decision  is  made  to  undergo  preliminary  design  (NRC, 
2008).  This  decision  point  is  called  Milestone  A  in  the  Defense  Acquisition  Framework. 
Following  a  successful  Milestone  A,  a  DoD  development  program  will  be  initiated  and 
one  or  more  prime  contractors  will  begin  a  preliminary  design  phase  to  demonstrate  fea¬ 
sibility,  reduce  risk,  and  refine  requirements.  The  program  formulation  elements  asso¬ 
ciated  with  the  Milestone  A  are  defined  in  DoD  5000  and  the  DAG  (DoD,  DoD 
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Instruction  5000.02,  2008),  but  additional  concept  maturity  elements  will  be  addressed 
herein. 

Information  Elements  for  Gate  3. 

In  both  industry  and  DoD,  an  investment  decision  must  be  made  based  upon  the 
information  and  analysis  contained  in  the  materiel  concept.  Much  of  this  information  is 
efficiently  and  effectively  conveyed  and  managed  via  architecture  products.  Although 
most  architecture  products  are  not  required  by  DoD  policy  until  the  later  Milestone  B  de¬ 
cision  to  enter  a  detailed  design  phase,  the  NRC  study  highlighted  several  important  ben¬ 
efits  to  earlier  development  of  systems  architecture  (NRC,  2008): 


1.  Architecture  can  mitigate  internal  and  external  system  complexity  risk  by  parti¬ 
tioning  the  system  into  separately  definable  and  procurable  parts. 

2.  Architecture  can  reduce  lifecycle  costs  through  the  process  of  breaking  down 
large  systems  into  more  easily  managed  components  whereby  potential  cost  and 
schedule  risks  can  be  identified. 

3.  The  construction  of  a  rigorous  systems  architecture  developed  early  in  the  pro¬ 
gram  will  aid  in  reducing  interface  complexity  control  problems  later  in  the  pro¬ 
gram  when  they  are  much  more  costly  to  fix  (NRC,  2008). 


Some  architecture  elements  created  in  earlier  phases  will  need  to  be  updated  and 
others  will  require  dramatic  additions  in  the  phase  preceding  the  concept  selection  deci¬ 
sion  (figure  5).  The  specifications  (MOP’s)  to  which  the  solution  will  be  measured  may 
require  updating.  An  initial  identification  of  characteristics  or  attributes  that  are  essential 
for  the  development  of  the  capability  should  be  conducted.  Mitigation  and  management 
plans  will  need  to  be  created  for  previously  identified  risks.  A  system  level  plan  to  de¬ 
velop  new  technologies  or  to  integrate  modified  technologies  should  also  be  detailed  for 
concept  selection.  DoD  calls  it  a  Systems  Engineering  Plan  (SEP)  and  uses  it  to  describe 
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the  process  of  technology  maturation 
(ODUSD(A&T)SSE/ED,  2008).  The  SEP  pro¬ 
vides  traceability  back  to  the  users’  needs  via 
elements  such  as  a  CONOPs,  risk  identification 
and  architecture  definition  eventually  focusing 
attention  towards  a  preferred  system  concept. 

The  final  version  of  the  overall  concept  should 
be  sufficient  to  characterize  how  the  solution  will 
meet  the  identified  need,  to  characterize  the 
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amount  of  risk  involved  with  developing  the  solu-  Figure  5  -  Information  Elements  for  Gate  3 
tion,  and  to  give  a  reasonable  idea  of  the  full  cost  associated  with  the  proposed  solution. 
This  work  describes  how  the  right  combination  of  architecture  views  along  with  the  other 
aforementioned  maturity  elements,  can  mature  a  concept  relative  to  the  early  decision 
points  in  the  development  process. 

An  Issue  of  Flexibility 


Every  development  project  must  pass  through  the  Material  Development  Decision 
but  from  this  gate,  the  Milestone  Decision  Authority  can  determine  what  phase  of  devel¬ 
opment  the  project  will  proceed  towards  (DoD,  DoD  Instruction  5000.02,  2008).  If  an 
MDA  decides  a  project  does  not  have  to  go  through  a  Milestone  A  gate  the  project  should 
still  perfonn  the  activities  to  develop  the  concept  associated  with  the  pre-Milestone  A 
phase.  Similarly,  any  solution  proposed  to  alleviate  a  deficiency  that  does  not  follow  the 
standard  development  process  should  perform  the  activities  to  understand  what  is  of  val¬ 
ue  to  the  user  and  how  the  user  might  use  the  proposed  solution.  The  reason  for  conduct- 
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ing  the  work  of  phases  that  a  project  might  “skip”  over  is  that  the  success  of  a  flexible 
development  process  is  “contingent  upon  ensuring  that  the  objectives  of  every  phase  of 


the  generic  process  are  met  and  that  critical  interactions  and  integration  issues  are  re¬ 
solved”  (Rafinejad,  2007,  p.  172). 
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Concept  Evaluation  and  Selection  Within  a  Stage-Gated  Process 


4-» 

c 

CL 

o 

cu 

4-» 

U 

u 

c 

cu 

o 

o 

cu 

LO 

Q. 

ClO 

_c 

cu 

c 

u 

cu 

c 

cu 

o 

o 

u 

LO 

(saior) 

1 

(ooosaoa) 

— i - 

(39dd) 

spaaN 

uopn|os 

saajnosay 

Figure  6  -  Concept  Maturity  Framework 
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(AFROC/JROC)  (MPA)  R  Hughes  2010  (MPA) 


The  Concept  Development  Process 

The  framework  developed  in  this  research  is  designed  to  act  as  a  benchmark  and 
common  language  for  those  evaluating  and  developing  material  concepts.  The  frame¬ 
work  is  designed  around  three  major  decision  gates  in  the  front  end  of  the  DoD  acquisi¬ 
tion  process  (figure  3).  The  information  recommended  at  each  gate  is  intended  to  charac¬ 
terize  the  needs  of  the  intended  users,  a  solution  to  meet  those  needs,  or  the  resources  re¬ 
quired  to  develop  the  solution.  The  decision  makers  must  determine  if  the  concept  before 
them  is  worthy  of  further  development  and  can  only  make  that  determination  if  they  are 
presented  with  the  right  type  and  amount  of  information. 

The  process  of  developing  a  concept  is  ordered  and  iterative.  As  shown  in  figure 
3,  information  is  developed  and  gathered  prior  to  each  decision  gate.  That  infonnation  is 
the  foundation  and  a  guide  for  the  development  activities  following  the  gate.  This  or¬ 
dered  activity  continues  through  the  phases  preceding  the  three  gates  with  the  activities  of 
each  subsequent  phase  building  upon  what  had  been  accomplished  (table  2).  However, 
this  phase  of  product  development  is  often  very  “fuzzy”  so,  there  is  an  iterative  aspect  of 
the  concept  development  process.  With  each  new  phase  comes  a  greater  amount  of  detail 
required,  which  could  bring  new  revelations.  Additionally,  circumstances  may  change 
during  the  course  of  development  that  may  alter  or  negate  the  work  accomplished  in  pre¬ 
vious  phases.  An  evaluation  should  be  conducted  to  determine  the  impacts  of  any 
changes  due  to  new  revelations  or  changing  circumstances. 

The  concept  maturity  framework  developed  in  this  research  was  used  as  a  bench¬ 
mark  similar  to  the  Child  Development  Inventory  (Ireton,  1995).  A  collaborative  re¬ 
search  project  was  conducted  with  Capt.  Daniel  Barker,  USAF,  to  validate  the  concept 
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maturity  framework.  Two  concepts,  in  different  phases  of  development,  were  assessed 
and  rated  not  based  upon  a  fully  developed  concept  but  relative  to  their  specific  phases  of 
development.  Additionally,  some  informational  maturity  elements,  of  one  of  the  con¬ 
cepts,  were  developed  to  further  mature  the  concept  (Barker,  2010). 


Table  2  -  Architecture  Views  by  Gate 
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V.  Conclusion,  Discussion  and  Recommendations 

Overview  of  the  Research 

The  process  of  transforming  an  idea  into  a  product  that  is  of  value  to  a  customer 
and  user  is  a  challenging  endeavor.  One  area  of  the  product  development  process  that  has 
a  great  impact  on  the  outcome  of  the  project  occurs  in  the  initial  phases:  the  development 
of  the  product  concept.  The  product  development  literature  agrees  on  the  importance  of 
beginning  a  development  project  with  a  complete  product  concept.  However,  the  defini¬ 
tion  of  what  a  complete  concept  is  was  not  discussed  in  the  literature.  This  research  at¬ 
tempted  to  define  the  Department  of  Defense  version  of  a  product  concept,  a  material 
concept,  and  develop  a  framework  made  up  of  informational  elements,  contained  in  the 
material  concept,  that  could  be  used  at  specific  decision  gates  within  the  DoD  acquisition 
process.  The  systems  architecture  views,  contained  within  the  Department  of  Defense 
Architecture  Framework  (DoDAF),  that  were  useful  in  presenting  the  information  were 
then  identified  and  incorporated  into  the  framework.  The  intended  purpose  of  this 
framework  was  to  serve  as  a  yardstick  for  decision  makers  to  detennine  if  a  concept  was 
complete  enough  for  the  next  phase  of  development.  Additionally,  the  framework  was 
intended  to  guide  those  developing  a  material  concept. 

Results  of  the  Research 

The  research  showed  that  a  developmental  scale  to  measure  a  product  concept 
could  be  created.  A  framework  was  developed  to  be  used  by  decision  makers  to  evaluate 
the  maturity  of  concepts  at  three  decision  gates  in  the  early  phases  of  the  product  devel- 
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opment  process  and  guide  those  developing  the  concepts  prior  to  the  decision  gates.  The 
information  contained  in  the  framework  was  based  upon  the  literature  and  the  purpose  of 
the  decision  gates.  DoDAF  views  were  then  selected  that  captured  the  infonnation  in  a 
manner  suitable  for  analysis  and  for  presentation. 

The  following  are  responses  to  the  research  questions  found  in  chapter  1 . 

Research  Question  1:  What  is  the  definition  of  a  product  concept  as  it  ap¬ 
plies  to  DoD  capability  development? 

The  literature  review  failed  to  identify  an  accepted  definition  of  a  product  concept 
in  either  the  literature  on  commercial  development  or  in  the  literature  on  DoD  acquisi¬ 
tion.  However,  the  literature  was  quite  consistent  about  the  purpose  of  a  product  concept 
and  what  type  of  information  should  be  found  in  a  product  concept.  For  the  purpose  of 
this  research  the  tenn  “material  concept”  was  given  to  the  DoD  equivalent  of  a  commer¬ 
cial  product  concept  and  was  defined  as  a  solution  that  meets  the  needs  of  the  customers 
and  users,  and  identification  of  the  resources  required  to  develop  the  solution. 

Research  Questions  2:  What  type  of  information  does  a  concept  require  to 
be  mature  in  a  stage-gated  process? 

At  the  concept  selection  gate,  a  concept  is  required  to  have  three  types  of  informa¬ 
tion.  First,  the  concept  must  identify  the  needs  of  an  intended  user  and  determine  what  is 
of  value  to  the  user.  Second,  the  concept  must  have  a  solution  to  that  need  and  show  that 
the  solution  will  meet  the  need.  Last,  the  concept  must  characterize  the  risk  associated 
with  developing  the  solution  and  the  resources  required  to  develop  the  solution.  These 
three  types  of  infonnation  will  be  collected  through  the  concept  development  process  and 
specific  “vehicles”  to  organize  that  information  were  identified. 
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Research  Question  3:  How  can  the  definition  of  a  mature  concept  account 
for  the  phases  of  development? 

The  type  and  amount  of  information  required  to  determine  if  a  material  concept 
was  mature  depended  upon  the  concept’s  phase  of  development.  A  type  of  developmen¬ 
tal  scale,  similar  to  one  used  to  evaluate  the  development  of  children  (Ireton,  1995),  can 
be  used  to  detennine  if  a  concept  is  mature  regardless  of  the  phase  of  development  as 
long  as  the  concept  is  evaluated  only  relative  to  where  it  should  be,  not  to  where  it  even¬ 
tually  must  be. 

Research  Question  4:  What  is  the  information  (criteria)  that  are  required  at 
each  gate  in  the  process? 

As  detailed  in  chapter  4,  and  represented  in  graphical  form  in  figure  3,  the  type  of 
information  and  the  amount  of  that  information  depended  upon  the  purpose  of  the  deci¬ 
sion  gate  following  the  phase  of  development.  At  the  first  gate,  the  AFROC  and  JROC 
are  faced  with  the  responsibility  of  validating  any  needed  capability  presented  within  an 
Initial  Capabilities  Document  (ICD)  and  confirming  that  the  capability  can  only  be  rea¬ 
lized  through  a  new  material  development.  The  ICD,  being  the  genesis  of  the  material 
concept,  must  show  what  the  users  need  and  why  they  cannot  accomplish  what  they  need 
with  existing  resources.  At  the  second  gate,  the  Material  Development  Decision  (MDD), 
the  Milestone  Decision  Authority  (MDA)  determines  if  a  project  will  continue  into  the 
acquisition  process  and  screens  potential  solutions  intended  to  meet  the  capability  need 
identified  in  the  ICD.  In  addition  to  the  ICD,  the  concept  must  now  have  at  least  one 
proposed  solution.  The  concept  will  have  to  show  how  the  proposed  solution  meets  the 
capability  need,  characterize  the  risk  associated  with  developing  the  solution  and  give  an 


initial  estimation  of  the  resources  required  to  develop  the  solution.  Between  the  second 
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gate  and  the  last  gate,  Milestone  A,  the  proposed  solutions  that  make  it  through  the  MDD 
will  need  to  be  developed  in  greater  detail.  The  different  concepts  will  need  to  be  com¬ 
peted  against  one  another  and  development  plans  will  need  to  be  created  for  the  remain¬ 
ing  phases  of  the  development  process  so,  the  MDA  can  select  a  concept  for  develop¬ 
ment. 

Research  Question  5:  What  architecture  views  present  the  information  re¬ 
quired  by  decision  makers  at  each  gate? 

DoDAF  System  Arcitecture  views  were  selected  for  each  gate  that  present  the  ne¬ 
cessary  infonnation.  The  views  associated  with  each  gate  can  be  located  in  chapter  4. 

Research  Question  6:  What  information,  if  any,  is  important  to  the  maturity 
of  a  concept  but  is  not  required  by  JCIDS  or  the  DoDI  5000  series? 

All  the  information  required  for  a  mature  concept  is  already  required  to  be  col¬ 
lected  by  JCIDS  and  the  DoDI  5000  series  but  it  is  required  much  later  in  the  develop¬ 
ment  process  than  it  should.  Many  of  the  architecture  views  found  in  the  concept  maturi¬ 
ty  framework  are  required  at  gates  that  follow  Milestone  A  but  the  research  indicates  that 
the  infonnation  contained  in  those  views  are  required  prior  to  Milestone  A  and  that  those 
views  should  be  created  before  concept  selection  and  developed  further  during  later  phas¬ 
es. 

Limitations  of  the  Research 

This  research  was  conducted  using  the  latest  guidance  and  instructions  regarding 
the  DoD  acquisition  process.  The  purpose  of  each  decision  gate  and  the  requirements  to 
pass  through  each  gate  were  determined  by  analyzing  these  guiding  documents.  Howev¬ 
er,  it  is  possible  that  the  purpose  and  function  of  the  decision  gates,  when  policy  is  put 
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into  action,  will  differ  from  what  is  in  the  guidance.  It  was  noted  in  a  previous  chapter 
that  ultimately,  the  decision  maker  is  the  authority  at  a  gate  but  this  framework  was 
created  without  the  input  of  any  who  hold  that  authority. 

The  framework  was  designed  around  the  DoD  acquisition  process  and  is  intended 
for  use  by  the  Department  of  Defense.  In  chapter  2  the  DoD  process  and  the  generic 
product  development  processes  were  compared  and  found  to  be  nearly  identical  in  pur¬ 
pose  and  function  but  additional  research  will  probably  be  required  before  any  attempt  to 
apply  the  framework  to  another  context. 

This  framework  was  developed  by  researchers  with  experience  in  the  DoD  acqui¬ 
sition  process  and  have  studied  the  product  development  process.  The  framework  was 
presented  to  domain  experts  and  their  feedback  was  incorporated  into  the  research.  How¬ 
ever,  relative  to  the  total  number  of  practitioners  ,  the  number  of  domain  experts  con¬ 
sulted  is  small.  Future  research  could  be  conducted  to  vet  the  framework  with  a  larger 
number  of  domain  experts. 

The  largest  limitation  of  this  research  is  that  the  framework  is  only  intended  to 
evaluate  the  completeness  of  the  concept  and  not  the  sufficiency  of  the  concept  to  meet 
the  needs  of  the  user.  It  is  necessary  for  a  concept  to  be  mature  to  pass  through  a  decision 
gate  successfully  but  it  is  not  sufficient.  Only  if  a  decision  maker  detennines  a  concept  is 
effective  and  suitable  to  meet  the  user  needs  will  it  proceed  to  the  next  phase  of  develop¬ 
ment. 
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Future  Research 


Through  the  course  of  this  research,  many  issues  were  identified  that  were  asso¬ 
ciated  with  DoD  system  development  but  were  outside  the  scope  of  this  research.  The 
following  are  future  research  topics  that  could  contribute  to  the  body  of  knowledge. 

First,  this  research  was  a  collaborative  effort  with  another  researcher  who  used  the 
framework  to  evaluate  two  concepts  relative  to  their  phase  of  development.  In  addition 
to  assessing  the  concepts,  efforts  were  made  to  further  mature  one  of  the  concepts  by  de¬ 
veloping  information  that  was  absent  in  the  concept  but  recommended  by  the  framework 
(Barker,  2010).  The  concepts  were  assessed  relative  to  the  criteria  for  the  JROC  and  the 
MDD  gates,  but  no  concept  was  assessed  relative  to  the  Milestone  A  gate.  Future  re¬ 
search  could  use  the  framework  to  evaluate  an  existing  material  concept  relative  to  the 
criteria  for  the  Milestone  A  gate.  This  research  could  validate  that  the  framework  cor¬ 
rectly  assesses  a  material  concept  at  each  gate  and  that  the  concept  could  be  matured  fur¬ 
ther  by  developing  the  infonnation  recommended  in  the  framework. 

A  second  area  of  research  could  be  conducted  in  the  area  of  the  cost  estimates  as¬ 
sociated  with  the  decision  gates.  A  framework  developed  for  “firms  associated  with  the 
manufacturing  and  production  of  chemicals,  petrochemicals,  and  hydrocarbon 
processing”  to  detennine  the  type  of  cost  estimate  based  upon  the  level  of  project  defini¬ 
tion  was  found  (AACE  International,  2005)  in  the  course  of  this  research  and  could  be 
adapted  to  the  “fuzzy  front-end”  of  DoD  capability  development. 

A  few  areas  of  potential  research  involve  personnel.  Currently,  within  the  Air 
Force,  the  Major  Commands  are  responsible  for  performing  the  analysis  that  supports  the 


63 


development  of  the  Initial  Capabilities  Document  (ICD).  The  people  performing  the 
tasks  during  these  early  phases  of  the  development  process  require  the  skills,  education, 
and  experience  to  perform  marketing  and  systems  engineering.  One  area  of  research 
could  look  at  the  skills,  education  and  experience  of  those  actually  perfonning  these 
tasks.  Also,  research  could  be  conducted  to  identify  how  much  interaction,  if  any,  the 
ICD  development  team  has  with  those  developing  the  solutions.  There  is  literature  that 
indicates  a  cross  functional  team  responsible  for  the  entire  development  of  the  product 
concept  is  a  good  practice  (Rafinejad,  2007). 

Final  Words  and  Takeaways 

A  development  project  started  with  a  complete  and  mature  product  concept  has 
the  necessary  foundation  for  a  successful  outcome.  A  complete  concept  identifies  the 
resources  necessary  to  develop  the  solution  and  characterizes  the  risk  involved  with  the 
development.  To  detennine  the  resources  and  understand  the  risk,  solutions  need  to  be 
developed  that  will  meet  the  needs  of  the  users.  In  order  to  propose  appropriate  solutions 
a  rigorous  analysis  of  user  needs  and  environment  is  required.  The  activities  conducted 
and  decisions  made  to  develop  the  product  concept  into  its  final  form  require  only  a  frac¬ 
tion  of  the  total  development  cost  yet,  can  determine  up  to  80%  of  a  system’s  life  cycle 
cost.  The  stage-gated  process  to  develop  and  select  a  product  concept  for  development 
must  ensure  only  well  developed  concepts  that  meet  the  needs  of  the  users  are  allowed  to 
pass  through  the  gates.  For  this  reason,  the  “go/kill/hold/recycle”  decision  gates  require  a 
robust  set  of  infonnation  criteria,  which  each  concept  should  meet  before  it  is  considered 
for  further  development.  However,  the  criteria  recommended  for  each  gate  in  the  concept 
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maturity  framework  will  only  benefit  the  development  process  if  the  “kill/hold/recycle” 
options  are  used  for  concepts  that  do  not  meet  the  necessary  criteria. 

A  final  takeaway  of  this  research  is  that  a  complete,  well-developed  concept  is 
necessary  but  it  is  not  sufficient  to  guarantee  success.  The  development  process  is  com¬ 
plex  and  requires  great  execution  as  well  as  great  planning.  The  intent  of  the  concept  ma¬ 
turity  framework  is  to  address  only  one  of  the  many  issues  experienced  in  DoD  concept 
development. 
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